Production Monitoring and Feedback Loops

Evaluation before release tells you how a system behaves on inputs you anticipated. Production tells you how it behaves on everything else. A Claude system that is not monitored on quality and safety as well as uptime is running blind, and the professional exam expects you to design the signals, the instrumentation and the loop that feeds learning back into the system.

The four signal families

Family	Signals	Why it matters
Quality	Sampled model-graded scores, user feedback, escalation and hand-off rate, task success, correction rate by humans downstream	Detects silent degradation after model, prompt or data changes
Safety	Refusal rate and categories, detected injection attempts, blocked tool calls, policy violations flagged by classifiers or hooks	Early warning of abuse and of over-refusal harming users
Reliability	Error rate by type (rate limit, overload, timeout, validation), latency percentiles, stream failures, tool error rates, retries	Standard operations plus model-specific failure modes
Cost	Tokens by type per route, cache hit rate, tool fees, turns per task, spend per workspace and tenant	Cost incidents are usually caching or loop regressions

Add drift across all four: changes in input distribution (languages, document types, lengths), output length, tool-call mix. Drift is often the first sign that users are doing something new or that an upstream system changed.

Instrumentation

• Trace every request with a trace ID that spans the gateway, each model call, each tool call and each subagent. OpenTelemetry is the usual carrier.
• Log the assembled request and final response (not streaming deltas) with prompt version, model ID, settings, usage and stop reason. Redact secrets and apply the same retention and access rules as any store of customer data.
• Emit structured events for agent runs: start, tool call, tool result summary, compaction, stop, cost.
• Capture user signals at the point of use: ratings, edits to generated text, whether a suggestion was accepted, whether the user escalated.
• Dashboards by route and segment, with alerts on cache hit rate, error rate, latency, refusal rate and cost anomalies.

Claude Code and the Agent SDK can export usage metrics and events through OpenTelemetry, which lets developer tooling feed the same observability stack as production services.

Sampled grading in production

You cannot grade every response, but you can grade a sample. Route a stratified sample of production responses to the same model grader used in evals, with the same rubric, and plot the score over time by segment. Send a smaller sample, plus low scores and user complaints, to human review. This gives a continuous quality metric that is comparable to your pre-release evals, which is what lets you detect regressions caused by changes you did not make, such as upstream data shifts.

Closing the loop

Monitoring that does not change anything is decoration. Define the loops explicitly:

1. Failures become cases. Every escalation, correction and low-graded sample is reviewed; confirmed failures are labelled and added to the regression dataset.
2. Patterns become prompt or tool changes. Clusters of failures drive changes, which go through CI gates like any change.
3. Human review labels calibrate graders. Agreement between human and model grades is tracked; disagreement triggers grader revision.
4. Model updates are shadowed. New model versions run in shadow against live traffic before promotion, graded with the production sampler.
5. Cost anomalies trigger profiling. A cache hit rate drop or turns-per-task rise opens an optimisation ticket.

Assign an owner to each loop. Unowned loops decay within a quarter.

Incident response for AI systems

AI incidents look different from outages: the system is up, fast and confidently wrong, or has been manipulated into an action. Prepare runbooks for: a quality regression after a change (roll back the bundle), a prompt injection that caused an action (disable the tool, audit affected sessions by trace ID, notify), a cost runaway (kill sessions, lower limits, investigate the loop), and an over-refusal spike (check policy changes and model updates). Each runbook starts with the trace and the bundle version, which is why both must be logged.

Privacy of monitoring data

Logs and samples are often the largest store of personal data in the system. Minimise what is logged, pseudonymise identifiers, apply retention limits, restrict who can view transcripts, and make sure model graders used on production samples run under the same data handling rules as the system itself.

Exam focus

• Monitor four families: quality, safety, reliability and cost, plus drift across all of them.
• Trace IDs must span gateway, model calls, tools and subagents; log assembled requests and responses with bundle version and usage, redacted and retention-governed.
• Sampled model grading in production provides a continuous quality metric comparable to pre-release evals.
• Feedback loops must be explicit and owned: failures become eval cases, patterns become changes, labels calibrate graders, model updates are shadowed.
• AI incidents are often "up but wrong"; runbooks start from the trace ID and the deployed bundle version.
• Monitoring data is personal data; minimise, pseudonymise and restrict access.