Operational runbook

SAIF.PremiumAudit.Batch runs unattended each night on BATCHPRD01 and posts premium adjustment transactions to the policy system. A failed or partial run delays premium accounting but is not customer-facing, so the standard response is to diagnose from the File0 and File2 streams and rerun the job with a restart checkpoint. Reruns are safe: audits are only marked Posted inside the same transaction that writes their PremiumTransaction row, so anything that failed is still Pending and will be picked up again.

Invocation

SAIF.PremiumAudit.Batch.exe Environment AuditDate [BatchSize] [RestartCheckpoint]

The scheduled task on BATCHPRD01 runs the equivalent of:

D:\Apps\PremiumAudit\SAIF.PremiumAudit.Batch.exe PROD 2026-02-17

Parameters

Position	Name	Example	Required	Notes
1	Environment	PROD	Yes	One of DEV, TEST, PROD. Selects the connection string and mail profile from App.config.
2	AuditDate	2026-02-17	Yes	Cut-off date in yyyy-MM-dd. The scheduled task passes the previous calendar day. Cannot be a future date.
3	BatchSize	1000	No	Maximum audits per run, 1-5000. Defaults to 1000. Lower it to 250 when working through a backlog during the business day.
4	RestartCheckpoint	884210	No	PolicyAuditId to resume from. Use the last checkpoint value written to File0 by the failed run.

Scheduling

Attribute	Value
Frequency	Daily
Typical start time	02:00
Expected duration	25-40 minutes
Scheduler job name	SAIF-PremiumAudit-Nightly (Windows Task Scheduler on BATCHPRD01)
Upstream dependencies	PayrollAuditExtract (completes by 01:30) and the nightly policy system replication job
Downstream dependents	PremiumBillingExtract (starts 04:00) and the premium accounting general ledger interface (starts 05:15)

Expected outputs

Stream	File pattern	Content	Retention
File0	D:\BatchLogs\PremiumAudit\PremiumAudit_File0_yyyyMMdd.log	Execution control log: header block, checkpoint lines every 100 audits, run totals, fatal errors	90 days on disk, then archived to \\saifarchive\batchlogs for 7 years
File1	D:\BatchLogs\PremiumAudit\PremiumAudit_File1_yyyyMMdd.rpt	Success detail report: one line per posted audit with policy number, audited payroll, recalculated premium, adjustment amount	90 days on disk, then archived for 7 years
File2	D:\BatchLogs\PremiumAudit\PremiumAudit_File2_yyyyMMdd.rpt	Error detail report: one line per rejected or failed audit with policy number, PolicyAuditId, business rule id, exception message	90 days on disk, then archived for 7 years

Environment configuration

Setting	Key	Purpose
Database	PolicyDbConnection (connection string, per environment)	SQL Server instance and database holding the dbo policy schema
Command timeout	SqlCommandTimeoutSeconds	Timeout applied to dbo.usp_GetPendingAudits and to each per-audit command. Default 120.
Log directory	LogDirectory	Root directory for the File0, File1, and File2 streams
Error threshold	ErrorThreshold	Per-audit failure count that stops the run and triggers notification. Default 25.
Notification recipients	NotificationRecipients	Semicolon-separated addresses for the catastrophic error email
SMTP host	SmtpHost	Relay used to send the notification email

Error scenarios

Scenario	Symptom	Impact	Recovery
Invalid parameters	ReturnCode = 1, usage text in the task's console output, File0 empty or missing	Job fails immediately, no audits processed	Open the SAIF-PremiumAudit-Nightly task definition and confirm the Arguments field reads PROD plus a yyyy-MM-dd audit date. Correct it and rerun the task for the same audit date.
Locked source file	ReturnCode = 2, File0 records IOException opening PremiumAudit_File1_yyyyMMdd.rpt	No audits processed; the run stops during PreProcess	The report from a prior run is usually still open in a viewer or being copied by the archive job. Identify the holder with handle.exe PremiumAudit_File1, close it or wait for the archive copy to finish, then rerun with the same arguments. If the file is genuinely orphaned, rename it to .locked and rerun; no data is lost because File1 is regenerated from the database.
Database timeout during a large batch	ReturnCode = 3 when dbo.usp_GetPendingAudits times out, or repeated SqlException: Timeout expired lines in File2 when individual audits time out	Nothing posted (ReturnCode 3), or a subset of audits left Pending	Confirm the policy database is responsive and check for blocking on PolicyAudit and PremiumTransaction. If the backlog is large, rerun with BatchSize of 250 and repeat until File0 reports zero audits read. Raise SqlCommandTimeoutSeconds only with DBA agreement; a persistent timeout on dbo.usp_GetPendingAudits normally means index maintenance is overdue.
Partial completion requiring restart	Task ends with ReturnCode 3, 4, or 5 and File0 shows a checkpoint line such as CHECKPOINT PolicyAuditId=884210 but no totals block	Audits after the checkpoint are unprocessed and still Pending; downstream PremiumBillingExtract will understate adjustments	Take the last PolicyAuditId from the File0 checkpoint line and rerun as SAIF.PremiumAudit.Batch.exe PROD 2026-02-17 1000 884210. Do not delete PremiumTransaction rows from the failed run: audits already posted are marked Posted and are excluded by dbo.usp_GetPendingAudits. Confirm the rerun's File0 totals plus the failed run's posted count equal the audits read by the first run.
Per-record failure	Lines in File2 with a business rule id such as BR-001, run still ends with ReturnCode 0	Individual audits unprocessed, remaining Pending	Send the File2 report to the underwriting audit queue. Validation rejections (BR-001, BR-002) need a data correction in the audit before the next nightly run; unexpected exceptions should be raised with the Policy Systems Team.
Catastrophic error threshold	ReturnCode = 5 and an email titled PremiumAudit batch stopped: error threshold exceeded	Batch stopped part way; downstream jobs run against incomplete data	Read the first ten File2 entries; a shared cause (a missing rate for a new class code, a schema change, an unavailable database) is usual. Fix the cause, then rerun from the File0 checkpoint. If it cannot be fixed before 04:00, notify the premium accounting on-call so PremiumBillingExtract can be held.

Monitoring and alerting

Alert	Trigger	Recipient	Severity
Premium Audit batch failed	Scheduled task last result is non-zero	Batch Operations (batchops@example.saif)	High
Premium Audit error threshold exceeded	Application email sent on ReturnCode 5	Policy Systems on-call (policysystems-oncall@example.saif)	High
Premium Audit did not complete by 03:30	No totals block in the File0 stream for the current date by 03:30	Batch Operations (batchops@example.saif)	Medium
Elevated audit rejection rate	File2 line count exceeds 10 percent of audits read	Underwriting Audit team (uwaudit@example.saif)	Low

Related documentation

• Batch lifecycle
• Operations and processes

Sources

• src/SAIF.PremiumAudit.Batch/Program.vb
• src/SAIF.PremiumAudit.Batch/AuditProcessor.vb
• src/SAIF.PremiumAudit.Batch/App.config