---
title: Operational runbook
summary: Deployment, health checks, logs, status codes, error scenarios, and recovery procedures.
status: draft
owner: Agency Systems Team
last_reviewed: 2026-09-10
---

# Operational runbook

`SAIF.Agency.Api` is a read-mostly service, so most incidents are read failures
that degrade a consumer rather than corrupt data: claims intake cannot resolve a
producer code, or the nightly `SAIF.Policy.RenewalNotice` run cannot name a
servicing agent and rejects those notices. The one write endpoint is protected
by an optimistic version check, so a retry after a failed contact update is
always safe — it either applies once or fails with `409`. The usual first
question is whether the service can reach the Oracle `AGENCY` schema.

## Deployment

| Attribute | Value |
| --- | --- |
| Host | IIS |
| Site / virtual directory | Site `agencyapi` on WEBPRD03 and WEBPRD04, virtual directory `/agency` |
| Application pool | `AgencyApiPool` (.NET CLR v4.0, Integrated pipeline) |
| Deployment mechanism | `azure-pipelines.yml`, stage `Deploy`, Web Deploy package pushed to both nodes in sequence |
| Artifact | `SAIF.Agency.Api.zip` produced by the `Build` stage |

The two PROD nodes sit behind the standard web load balancer and are deployed
one at a time, so a deployment does not take the service offline.

## Environments

| Environment | Base address | Environment track | Notes |
| --- | --- | --- | --- |
| DEV | `https://agencyapi-dev.example.saif/agency` | `dev` | Single node, WEBDEV01. Swagger UI reachable. |
| TEST | `https://agencyapi-test.example.saif/agency` | `test` | Single node, WEBTST01. Refreshed from a masked copy of the PROD `AGENCY` schema each Sunday. |
| PROD | `https://agencyapi.example.saif/agency` | `prod` | Two nodes, WEBPRD03 and WEBPRD04, behind the web load balancer. |

## Health verification

The service exposes no health endpoint. Use the lightest read in the catalog: a
lookup of a known stable agency number exercises routing, authorization, the
Autofac container, and the Oracle connection in one call.

| Check | Request | Expected result |
| --- | --- | --- |
| Service responds and can reach Oracle | `GET /v1/agencies/104782` | `200 OK` with an `AgencyDetailDto` for Cascade Risk Partners LLC |
| Attribute routing is registered | `GET /v1/agencies?producerCode=CRP01` | `200 OK` with a JSON array; a `404` here means `MapHttpAttributeRoutes()` did not run |
| API documentation surface reachable | `GET /swagger/ui/index` | Swashbuckle UI listing all ten endpoints |

## Status codes

| Code | Meaning | Operator action |
| --- | --- | --- |
| `200 OK` | Read succeeded; an empty array is a valid search result | None |
| `204 No Content` | Contact update applied; `X-Agency-Version` carries the new version | None |
| `400 Bad Request` | Caller supplied a malformed agency number, an unbounded search, or invalid contact details; produced by the action or by `AgencyApiExceptionFilter` | Return the `code` and `details` to the caller; no server-side action |
| `401 Unauthorized` | IIS rejected an unauthenticated caller, or `SaifAuthorize` rejected a caller without `AgencyRead` or `AgencyContactWrite` | Confirm the consumer's service account and its role membership; a recent application pool identity change is the usual cause |
| `404 Not Found` | No route matched (framework), or the agency does not exist or is terminated (action) | Check the log entry: a framework `404` has no correlation id and means the route is wrong; an action `404` carries `code` `AGENCY_NOT_FOUND` and means the identifier is wrong |
| `409 Conflict` | Contact update refused because the agency is not active, or because the caller's `expectedVersion` was stale; produced by `AgencyApiExceptionFilter` | Expected during concurrent portal edits. Advise the caller to re-read the agency and resubmit. Persistent `AGENCY_NOT_ACTIVE` means the agency status is genuinely not `Active`. |
| `500 Internal Server Error` | Unhandled exception, most often Oracle unavailable or a command timeout | Escalate with the correlation id from the response body; the matching log entry carries the full exception |

## Logs

| Surface | Location | Content | Retention |
| --- | --- | --- | --- |
| `ILogService` | Oracle table `SAIF_LOG.APPLICATION_LOG`, `APPLICATION = 'SAIF.Agency.Api'` | One row per request with correlation id, route, caller, elapsed milliseconds, and status; full exception detail for every `500` | 90 days, purged by the `SAIF_LOG` housekeeping job |
| IIS request logs | `D:\inetpub\logs\LogFiles\W3SVC2\` on WEBPRD03 and WEBPRD04 | Request line, status, and time taken, including requests rejected by IIS before managed code runs | 30 days on disk |
| Windows event log | Application log on each web node | Application pool start-up failures, including a failure in `AutofacConfig.Register` or the Environment API read | Server default |
| OpenTelemetry traces | Not determined from source — the exporter endpoint is supplied by the platform, not by this repository | Distributed traces tagged with owner `Agency Systems Team` | Not determined from source |

## Environment configuration

| Setting | Key | Purpose |
| --- | --- | --- |
| Base address | `AgencyApi.BaseUrl` | Address consumers resolve for this environment's track |
| Database | `agencyDb` | Oracle connection resolved at start-up through the Environment API; schema `AGENCY` |
| Command timeout | `AgencyApi.CommandTimeoutSeconds` | Timeout applied to every NHibernate query. Default `30`. |
| Swagger UI | `AgencyApi.SwaggerEnabled` | Enables the Swashbuckle UI; `true` in every environment, reachable only from the internal network |
| Log level | `AgencyApi.LogLevel` | `Info` in PROD, `Debug` in DEV and TEST |

See Configuration for the complete settings catalog, including the runtime resolution of database connections.

## Error scenarios

| Scenario | Symptom | Impact | Recovery |
| --- | --- | --- | --- |
| Environment API unavailable | Application pool fails to start; Windows event log records the failure inside `AutofacConfig.Register`; every request returns `503` from IIS | Total outage on the affected node | Verify Environment API availability for the node's track, then recycle `AgencyApiPool`. Configuration is read only at start-up, so a node that started before the outage keeps serving. |
| Oracle connection resolution fails | `500` responses with `code` `INTERNAL_ERROR`; `APPLICATION_LOG` shows `ORA-12541` or `ORA-01017` from the first data-backed request | Every endpoint fails; consumers degrade as described below | Confirm the `agencyDb` key resolves for this track and that the service account's Oracle password has not expired, then confirm the database is listening. No restart is needed once the database is reachable. |
| Oracle command timeout on search | Intermittent `500` responses confined to `GET /v1/agencies`; `APPLICATION_LOG` shows the 30 second timeout on the producer code query | Claims intake cannot resolve producer codes; lookups by agency number keep working | Check for blocking on `AGENCY` and for missing statistics on `AGENCY_PRODUCER_CODE`. Raise `AgencyApi.CommandTimeoutSeconds` only with DBA agreement — a persistent timeout on this query means index maintenance is overdue. |
| Renewal notice run cannot resolve agents | `SAIF.Policy.RenewalNotice` exits with code 6 and its File2 report lists policies with no servicing agent | Renewal notices for the affected policies are not queued for delivery that night | Confirm this service is healthy, then rerun the notice job for the same notice date. The job is restartable and will not duplicate notices already queued. |
| Concurrent contact edits | Repeated `409 AGENCY_VERSION_STALE` from the agency portal for one agency | One user's edit is rejected; no data is lost or partially written | Expected behaviour, not a fault. Advise the user to reload the agency and reapply the change. Escalate only if the version never advances, which would indicate the update is failing to commit. |
| Application pool recycled under load | A burst of `503` responses on one node, then recovery within a few seconds | Consumers see transient failures; the load balancer keeps the other node serving | Check the recycle reason in the event log. A private memory recycle during the nightly notice run is the common cause; move the recycle window rather than raising the limit. |
| Deployment applied to only one node | Inconsistent responses between calls — a new field present on some responses and absent on others | Consumers see intermittent contract differences | Confirm the `Deploy` stage completed for both WEBPRD03 and WEBPRD04, and redeploy the node left behind. |

## Monitoring and alerting

| Alert | Trigger | Recipient | Severity |
| --- | --- | --- | --- |
| Agency API availability | The synthetic probe `GET /v1/agencies/104782` fails twice in five minutes | Agency Systems on-call (`agencysystems-oncall@example.saif`) | High |
| Agency API error rate | `5xx` responses exceed 2 percent of requests over 15 minutes, from the OpenTelemetry traces tagged with owner `Agency Systems Team` | Agency Systems on-call (`agencysystems-oncall@example.saif`) | High |
| Agency API latency | 95th percentile request duration exceeds 2 seconds over 15 minutes | Agency Systems Team (`agencysystems@example.saif`) | Medium |
| Unauthorized request spike | More than 50 `401` responses in 15 minutes | Agency Systems Team (`agencysystems@example.saif`) | Low |

## Related documentation
- [API index](index.md)
- [Overview](overview.md)
- [Architecture](architecture.md)
- [Endpoints](endpoints.md)
- [Authentication and authorization](authentication-authorization.md)
- Dependencies
- Configuration
- Operations and processes
- [Business rules](business-rules.md)
- Data model
- Tech debt

## Sources
- `azure-pipelines.yml`
- `src/SAIF.Agency.Api/Web.config`
- `src/SAIF.Agency.Api/Global.asax.cs`
- `src/SAIF.Agency.Api/App_Start/SwaggerConfig.cs`
- `src/SAIF.Agency.Api/Filters/AgencyApiExceptionFilter.cs`
- `src/SAIF.Agency.Api.Services/AgencyService.cs`