Example: API operational runbook

operational-runbook.md is what an operator reads at 2am. It records where the service is deployed, how to prove it is alive, what each status code means for them, where the logs are, and what to do about the failures this service actually produces. Doc gen produces one per API at docs/apis/<api-slug>/operational-runbook.md.

What reviewers should check

• Health verification uses a real endpoint from the catalog. Most Web API 2 services have no health route; inventing GET /health sends an operator to a 404 during an incident.
• Deployment rows are evidenced by a pipeline, script, or comment. "Not determined from source" is a better answer than a plausible site name.
• The status code table lists the codes this service really produces — including any 409 it raises — and each row gives the operator an action, not a restatement of the HTTP spec.
• Log destinations are real: a path, a table, or a sink named in configuration. When the sink is configured outside the repository, the row says so.
• Error scenarios name a symptom an operator can actually observe (a status code, a log entry, a failing consumer) and a recovery that someone without source access can carry out.
• Recovery text is specific about data safety — whether a retry is safe, and what a partially applied write leaves behind.
• Monitoring describes telemetry that is configured. When none is, the document says so rather than describing a hypothetical alert.

Example

The document below is a complete, correct api-operational-runbook.md for a fictional API.

View rendered example — the same document as a real page, so you can compare rendering as well as source.

---
title: Operational runbook
summary: Deployment, health checks, logs, status codes, error scenarios, and recovery procedures.
status: draft
owner: Agency Systems Team
last_reviewed: 2026-09-10
---

# Operational runbook

`SAIF.Agency.Api` is a read-mostly service, so most incidents are read failures
that degrade a consumer rather than corrupt data: claims intake cannot resolve a
producer code, or the nightly `SAIF.Policy.RenewalNotice` run cannot name a
servicing agent and rejects those notices. The one write endpoint is protected
by an optimistic version check, so a retry after a failed contact update is
always safe — it either applies once or fails with `409`. The usual first
question is whether the service can reach the Oracle `AGENCY` schema.

## Deployment

| Attribute | Value |
| --- | --- |
| Host | IIS |
| Site / virtual directory | Site `agencyapi` on WEBPRD03 and WEBPRD04, virtual directory `/agency` |
| Application pool | `AgencyApiPool` (.NET CLR v4.0, Integrated pipeline) |
| Deployment mechanism | `azure-pipelines.yml`, stage `Deploy`, Web Deploy package pushed to both nodes in sequence |
| Artifact | `SAIF.Agency.Api.zip` produced by the `Build` stage |

The two PROD nodes sit behind the standard web load balancer and are deployed
one at a time, so a deployment does not take the service offline.

## Environments

| Environment | Base address | Environment track | Notes |
| --- | --- | --- | --- |
| DEV | `https://agencyapi-dev.example.saif/agency` | `dev` | Single node, WEBDEV01. Swagger UI reachable. |
| TEST | `https://agencyapi-test.example.saif/agency` | `test` | Single node, WEBTST01. Refreshed from a masked copy of the PROD `AGENCY` schema each Sunday. |
| PROD | `https://agencyapi.example.saif/agency` | `prod` | Two nodes, WEBPRD03 and WEBPRD04, behind the web load balancer. |

## Health verification

The service exposes no health endpoint. Use the lightest read in the catalog: a
lookup of a known stable agency number exercises routing, authorization, the
Autofac container, and the Oracle connection in one call.

| Check | Request | Expected result |
| --- | --- | --- |
| Service responds and can reach Oracle | `GET /v1/agencies/104782` | `200 OK` with an `AgencyDetailDto` for Cascade Risk Partners LLC |
| Attribute routing is registered | `GET /v1/agencies?producerCode=CRP01` | `200 OK` with a JSON array; a `404` here means `MapHttpAttributeRoutes()` did not run |
| API documentation surface reachable | `GET /swagger/ui/index` | Swashbuckle UI listing all ten endpoints |

## Status codes

| Code | Meaning | Operator action |
| --- | --- | --- |
| `200 OK` | Read succeeded; an empty array is a valid search result | None |
| `204 No Content` | Contact update applied; `X-Agency-Version` carries the new version | None |
| `400 Bad Request` | Caller supplied a malformed agency number, an unbounded search, or invalid contact details; produced by the action or by `AgencyApiExceptionFilter` | Return the `code` and `details` to the caller; no server-side action |
| `401 Unauthorized` | IIS rejected an unauthenticated caller, or `SaifAuthorize` rejected a caller without `AgencyRead` or `AgencyContactWrite` | Confirm the consumer's service account and its role membership; a recent application pool identity change is the usual cause |
| `404 Not Found` | No route matched (framework), or the agency does not exist or is terminated (action) | Check the log entry: a framework `404` has no correlation id and means the route is wrong; an action `404` carries `code` `AGENCY_NOT_FOUND` and means the identifier is wrong |
| `409 Conflict` | Contact update refused because the agency is not active, or because the caller's `expectedVersion` was stale; produced by `AgencyApiExceptionFilter` | Expected during concurrent portal edits. Advise the caller to re-read the agency and resubmit. Persistent `AGENCY_NOT_ACTIVE` means the agency status is genuinely not `Active`. |
| `500 Internal Server Error` | Unhandled exception, most often Oracle unavailable or a command timeout | Escalate with the correlation id from the response body; the matching log entry carries the full exception |

## Logs

| Surface | Location | Content | Retention |
| --- | --- | --- | --- |
| `ILogService` | Oracle table `SAIF_LOG.APPLICATION_LOG`, `APPLICATION = 'SAIF.Agency.Api'` | One row per request with correlation id, route, caller, elapsed milliseconds, and status; full exception detail for every `500` | 90 days, purged by the `SAIF_LOG` housekeeping job |
| IIS request logs | `D:\inetpub\logs\LogFiles\W3SVC2\` on WEBPRD03 and WEBPRD04 | Request line, status, and time taken, including requests rejected by IIS before managed code runs | 30 days on disk |
| Windows event log | Application log on each web node | Application pool start-up failures, including a failure in `AutofacConfig.Register` or the Environment API read | Server default |
| OpenTelemetry traces | Not determined from source — the exporter endpoint is supplied by the platform, not by this repository | Distributed traces tagged with owner `Agency Systems Team` | Not determined from source |

## Environment configuration

| Setting | Key | Purpose |
| --- | --- | --- |
| Base address | `AgencyApi.BaseUrl` | Address consumers resolve for this environment's track |
| Database | `agencyDb` | Oracle connection resolved at start-up through the Environment API; schema `AGENCY` |
| Command timeout | `AgencyApi.CommandTimeoutSeconds` | Timeout applied to every NHibernate query. Default `30`. |
| Swagger UI | `AgencyApi.SwaggerEnabled` | Enables the Swashbuckle UI; `true` in every environment, reachable only from the internal network |
| Log level | `AgencyApi.LogLevel` | `Info` in PROD, `Debug` in DEV and TEST |

See [Configuration](configuration.md) for the complete settings catalog, including the runtime resolution of database connections.

## Error scenarios

| Scenario | Symptom | Impact | Recovery |
| --- | --- | --- | --- |
| Environment API unavailable | Application pool fails to start; Windows event log records the failure inside `AutofacConfig.Register`; every request returns `503` from IIS | Total outage on the affected node | Verify Environment API availability for the node's track, then recycle `AgencyApiPool`. Configuration is read only at start-up, so a node that started before the outage keeps serving. |
| Oracle connection resolution fails | `500` responses with `code` `INTERNAL_ERROR`; `APPLICATION_LOG` shows `ORA-12541` or `ORA-01017` from the first data-backed request | Every endpoint fails; consumers degrade as described below | Confirm the `agencyDb` key resolves for this track and that the service account's Oracle password has not expired, then confirm the database is listening. No restart is needed once the database is reachable. |
| Oracle command timeout on search | Intermittent `500` responses confined to `GET /v1/agencies`; `APPLICATION_LOG` shows the 30 second timeout on the producer code query | Claims intake cannot resolve producer codes; lookups by agency number keep working | Check for blocking on `AGENCY` and for missing statistics on `AGENCY_PRODUCER_CODE`. Raise `AgencyApi.CommandTimeoutSeconds` only with DBA agreement — a persistent timeout on this query means index maintenance is overdue. |
| Renewal notice run cannot resolve agents | `SAIF.Policy.RenewalNotice` exits with code 6 and its File2 report lists policies with no servicing agent | Renewal notices for the affected policies are not queued for delivery that night | Confirm this service is healthy, then rerun the notice job for the same notice date. The job is restartable and will not duplicate notices already queued. |
| Concurrent contact edits | Repeated `409 AGENCY_VERSION_STALE` from the agency portal for one agency | One user's edit is rejected; no data is lost or partially written | Expected behaviour, not a fault. Advise the user to reload the agency and reapply the change. Escalate only if the version never advances, which would indicate the update is failing to commit. |
| Application pool recycled under load | A burst of `503` responses on one node, then recovery within a few seconds | Consumers see transient failures; the load balancer keeps the other node serving | Check the recycle reason in the event log. A private memory recycle during the nightly notice run is the common cause; move the recycle window rather than raising the limit. |
| Deployment applied to only one node | Inconsistent responses between calls — a new field present on some responses and absent on others | Consumers see intermittent contract differences | Confirm the `Deploy` stage completed for both WEBPRD03 and WEBPRD04, and redeploy the node left behind. |

## Monitoring and alerting

| Alert | Trigger | Recipient | Severity |
| --- | --- | --- | --- |
| Agency API availability | The synthetic probe `GET /v1/agencies/104782` fails twice in five minutes | Agency Systems on-call (`agencysystems-oncall@example.saif`) | High |
| Agency API error rate | `5xx` responses exceed 2 percent of requests over 15 minutes, from the OpenTelemetry traces tagged with owner `Agency Systems Team` | Agency Systems on-call (`agencysystems-oncall@example.saif`) | High |
| Agency API latency | 95th percentile request duration exceeds 2 seconds over 15 minutes | Agency Systems Team (`agencysystems@example.saif`) | Medium |
| Unauthorized request spike | More than 50 `401` responses in 15 minutes | Agency Systems Team (`agencysystems@example.saif`) | Low |

## Related documentation
- [API index](index.md)
- [Overview](overview.md)
- [Architecture](architecture.md)
- [Endpoints](endpoints.md)
- [Authentication and authorization](authentication-authorization.md)
- [Dependencies](dependencies.md)
- [Configuration](configuration.md)
- [Operations and processes](operations-processes.md)
- [Business rules](business-rules.md)
- [Data model](data-model.md)
- [Tech debt](tech-debt.md)

## Sources
- `azure-pipelines.yml`
- `src/SAIF.Agency.Api/Web.config`
- `src/SAIF.Agency.Api/Global.asax.cs`
- `src/SAIF.Agency.Api/App_Start/SwaggerConfig.cs`
- `src/SAIF.Agency.Api/Filters/AgencyApiExceptionFilter.cs`
- `src/SAIF.Agency.Api.Services/AgencyService.cs`