---
title: Operational runbook
summary: Parameters, scheduling, expected outputs, error scenarios, and recovery procedures.
status: draft
owner: Policy Systems Team
last_reviewed: 2026-02-18
---

# Operational runbook

`SAIF.PremiumAudit.Batch` runs unattended each night on BATCHPRD01 and posts
premium adjustment transactions to the policy system. A failed or partial run
delays premium accounting but is not customer-facing, so the standard response
is to diagnose from the File0 and File2 streams and rerun the job with a restart
checkpoint. Reruns are safe: audits are only marked `Posted` inside the same
transaction that writes their `PremiumTransaction` row, so anything that failed
is still `Pending` and will be picked up again.

## Invocation

```
SAIF.PremiumAudit.Batch.exe Environment AuditDate [BatchSize] [RestartCheckpoint]
```

The scheduled task on BATCHPRD01 runs the equivalent of:

```
D:\Apps\PremiumAudit\SAIF.PremiumAudit.Batch.exe PROD 2026-02-17
```

### Parameters

| Position | Name | Example | Required | Notes |
| --- | --- | --- | --- | --- |
| 1 | Environment | `PROD` | Yes | One of `DEV`, `TEST`, `PROD`. Selects the connection string and mail profile from `App.config`. |
| 2 | AuditDate | `2026-02-17` | Yes | Cut-off date in `yyyy-MM-dd`. The scheduled task passes the previous calendar day. Cannot be a future date. |
| 3 | BatchSize | `1000` | No | Maximum audits per run, 1-5000. Defaults to `1000`. Lower it to `250` when working through a backlog during the business day. |
| 4 | RestartCheckpoint | `884210` | No | `PolicyAuditId` to resume from. Use the last checkpoint value written to File0 by the failed run. |

## Scheduling

| Attribute | Value |
| --- | --- |
| Frequency | Daily |
| Typical start time | 02:00 |
| Expected duration | 25-40 minutes |
| Scheduler job name | `SAIF-PremiumAudit-Nightly` (Windows Task Scheduler on BATCHPRD01) |
| Upstream dependencies | `PayrollAuditExtract` (completes by 01:30) and the nightly policy system replication job |
| Downstream dependents | `PremiumBillingExtract` (starts 04:00) and the premium accounting general ledger interface (starts 05:15) |

## Expected outputs

| Stream | File pattern | Content | Retention |
| --- | --- | --- | --- |
| File0 | `D:\BatchLogs\PremiumAudit\PremiumAudit_File0_yyyyMMdd.log` | Execution control log: header block, checkpoint lines every 100 audits, run totals, fatal errors | 90 days on disk, then archived to `\\saifarchive\batchlogs` for 7 years |
| File1 | `D:\BatchLogs\PremiumAudit\PremiumAudit_File1_yyyyMMdd.rpt` | Success detail report: one line per posted audit with policy number, audited payroll, recalculated premium, adjustment amount | 90 days on disk, then archived for 7 years |
| File2 | `D:\BatchLogs\PremiumAudit\PremiumAudit_File2_yyyyMMdd.rpt` | Error detail report: one line per rejected or failed audit with policy number, `PolicyAuditId`, business rule id, exception message | 90 days on disk, then archived for 7 years |

## Environment configuration

| Setting | Key | Purpose |
| --- | --- | --- |
| Database | `PolicyDbConnection` (connection string, per environment) | SQL Server instance and database holding the `dbo` policy schema |
| Command timeout | `SqlCommandTimeoutSeconds` | Timeout applied to `dbo.usp_GetPendingAudits` and to each per-audit command. Default `120`. |
| Log directory | `LogDirectory` | Root directory for the File0, File1, and File2 streams |
| Error threshold | `ErrorThreshold` | Per-audit failure count that stops the run and triggers notification. Default `25`. |
| Notification recipients | `NotificationRecipients` | Semicolon-separated addresses for the catastrophic error email |
| SMTP host | `SmtpHost` | Relay used to send the notification email |

## Error scenarios

| Scenario | Symptom | Impact | Recovery |
| --- | --- | --- | --- |
| Invalid parameters | ReturnCode = 1, usage text in the task's console output, File0 empty or missing | Job fails immediately, no audits processed | Open the `SAIF-PremiumAudit-Nightly` task definition and confirm the Arguments field reads `PROD` plus a `yyyy-MM-dd` audit date. Correct it and rerun the task for the same audit date. |
| Locked source file | ReturnCode = 2, File0 records `IOException` opening `PremiumAudit_File1_yyyyMMdd.rpt` | No audits processed; the run stops during PreProcess | The report from a prior run is usually still open in a viewer or being copied by the archive job. Identify the holder with `handle.exe PremiumAudit_File1`, close it or wait for the archive copy to finish, then rerun with the same arguments. If the file is genuinely orphaned, rename it to `.locked` and rerun; no data is lost because File1 is regenerated from the database. |
| Database timeout during a large batch | ReturnCode = 3 when `dbo.usp_GetPendingAudits` times out, or repeated `SqlException: Timeout expired` lines in File2 when individual audits time out | Nothing posted (ReturnCode 3), or a subset of audits left `Pending` | Confirm the policy database is responsive and check for blocking on `PolicyAudit` and `PremiumTransaction`. If the backlog is large, rerun with `BatchSize` of `250` and repeat until File0 reports zero audits read. Raise `SqlCommandTimeoutSeconds` only with DBA agreement; a persistent timeout on `dbo.usp_GetPendingAudits` normally means index maintenance is overdue. |
| Partial completion requiring restart | Task ends with ReturnCode 3, 4, or 5 and File0 shows a checkpoint line such as `CHECKPOINT PolicyAuditId=884210` but no totals block | Audits after the checkpoint are unprocessed and still `Pending`; downstream `PremiumBillingExtract` will understate adjustments | Take the last `PolicyAuditId` from the File0 checkpoint line and rerun as `SAIF.PremiumAudit.Batch.exe PROD 2026-02-17 1000 884210`. Do not delete `PremiumTransaction` rows from the failed run: audits already posted are marked `Posted` and are excluded by `dbo.usp_GetPendingAudits`. Confirm the rerun's File0 totals plus the failed run's posted count equal the audits read by the first run. |
| Per-record failure | Lines in File2 with a business rule id such as `BR-001`, run still ends with ReturnCode 0 | Individual audits unprocessed, remaining `Pending` | Send the File2 report to the underwriting audit queue. Validation rejections (`BR-001`, `BR-002`) need a data correction in the audit before the next nightly run; unexpected exceptions should be raised with the Policy Systems Team. |
| Catastrophic error threshold | ReturnCode = 5 and an email titled `PremiumAudit batch stopped: error threshold exceeded` | Batch stopped part way; downstream jobs run against incomplete data | Read the first ten File2 entries; a shared cause (a missing rate for a new class code, a schema change, an unavailable database) is usual. Fix the cause, then rerun from the File0 checkpoint. If it cannot be fixed before 04:00, notify the premium accounting on-call so `PremiumBillingExtract` can be held. |

## Monitoring and alerting

| Alert | Trigger | Recipient | Severity |
| --- | --- | --- | --- |
| Premium Audit batch failed | Scheduled task last result is non-zero | Batch Operations (`batchops@example.saif`) | High |
| Premium Audit error threshold exceeded | Application email sent on ReturnCode 5 | Policy Systems on-call (`policysystems-oncall@example.saif`) | High |
| Premium Audit did not complete by 03:30 | No totals block in the File0 stream for the current date by 03:30 | Batch Operations (`batchops@example.saif`) | Medium |
| Elevated audit rejection rate | File2 line count exceeds 10 percent of audits read | Underwriting Audit team (`uwaudit@example.saif`) | Low |

## Related documentation
- [Batch lifecycle](batch-lifecycle.md)
- Operations and processes

## Sources
- `src/SAIF.PremiumAudit.Batch/Program.vb`
- `src/SAIF.PremiumAudit.Batch/AuditProcessor.vb`
- `src/SAIF.PremiumAudit.Batch/App.config`