SAIF.Agency.Api is a read-mostly service, so most incidents are read failures
that degrade a consumer rather than corrupt data: claims intake cannot resolve a
producer code, or the nightly SAIF.Policy.RenewalNotice run cannot name a
servicing agent and rejects those notices. The one write endpoint is protected
by an optimistic version check, so a retry after a failed contact update is
always safe — it either applies once or fails with 409. The usual first
question is whether the service can reach the Oracle AGENCY schema.
The service exposes no health endpoint. Use the lightest read in the catalog: a
lookup of a known stable agency number exercises routing, authorization, the
Autofac container, and the Oracle connection in one call.
Check
Request
Expected result
Service responds and can reach Oracle
GET /v1/agencies/104782
200 OK with an AgencyDetailDto for Cascade Risk Partners LLC
Attribute routing is registered
GET /v1/agencies?producerCode=CRP01
200 OK with a JSON array; a 404 here means MapHttpAttributeRoutes() did not run
Read succeeded; an empty array is a valid search result
None
204 No Content
Contact update applied; X-Agency-Version carries the new version
None
400 Bad Request
Caller supplied a malformed agency number, an unbounded search, or invalid contact details; produced by the action or by AgencyApiExceptionFilter
Return the code and details to the caller; no server-side action
401 Unauthorized
IIS rejected an unauthenticated caller, or SaifAuthorize rejected a caller without AgencyRead or AgencyContactWrite
Confirm the consumer's service account and its role membership; a recent application pool identity change is the usual cause
404 Not Found
No route matched (framework), or the agency does not exist or is terminated (action)
Check the log entry: a framework 404 has no correlation id and means the route is wrong; an action 404 carries codeAGENCY_NOT_FOUND and means the identifier is wrong
409 Conflict
Contact update refused because the agency is not active, or because the caller's expectedVersion was stale; produced by AgencyApiExceptionFilter
Expected during concurrent portal edits. Advise the caller to re-read the agency and resubmit. Persistent AGENCY_NOT_ACTIVE means the agency status is genuinely not Active.
500 Internal Server Error
Unhandled exception, most often Oracle unavailable or a command timeout
Escalate with the correlation id from the response body; the matching log entry carries the full exception
Application pool fails to start; Windows event log records the failure inside AutofacConfig.Register; every request returns 503 from IIS
Total outage on the affected node
Verify Environment API availability for the node's track, then recycle AgencyApiPool. Configuration is read only at start-up, so a node that started before the outage keeps serving.
Oracle connection resolution fails
500 responses with codeINTERNAL_ERROR; APPLICATION_LOG shows ORA-12541 or ORA-01017 from the first data-backed request
Every endpoint fails; consumers degrade as described below
Confirm the agencyDb key resolves for this track and that the service account's Oracle password has not expired, then confirm the database is listening. No restart is needed once the database is reachable.
Oracle command timeout on search
Intermittent 500 responses confined to GET /v1/agencies; APPLICATION_LOG shows the 30 second timeout on the producer code query
Claims intake cannot resolve producer codes; lookups by agency number keep working
Check for blocking on AGENCY and for missing statistics on AGENCY_PRODUCER_CODE. Raise AgencyApi.CommandTimeoutSeconds only with DBA agreement — a persistent timeout on this query means index maintenance is overdue.
Renewal notice run cannot resolve agents
SAIF.Policy.RenewalNotice exits with code 6 and its File2 report lists policies with no servicing agent
Renewal notices for the affected policies are not queued for delivery that night
Confirm this service is healthy, then rerun the notice job for the same notice date. The job is restartable and will not duplicate notices already queued.
Concurrent contact edits
Repeated 409 AGENCY_VERSION_STALE from the agency portal for one agency
One user's edit is rejected; no data is lost or partially written
Expected behaviour, not a fault. Advise the user to reload the agency and reapply the change. Escalate only if the version never advances, which would indicate the update is failing to commit.
Application pool recycled under load
A burst of 503 responses on one node, then recovery within a few seconds
Consumers see transient failures; the load balancer keeps the other node serving
Check the recycle reason in the event log. A private memory recycle during the nightly notice run is the common cause; move the recycle window rather than raising the limit.
Deployment applied to only one node
Inconsistent responses between calls — a new field present on some responses and absent on others
Consumers see intermittent contract differences
Confirm the Deploy stage completed for both WEBPRD03 and WEBPRD04, and redeploy the node left behind.