Observability, Evaluation and Regression Testing

A Claude system that cannot be measured cannot be improved safely. Observability tells you what the system did in production; evaluation tells you how well it does on cases you care about; regression testing stops a change to a prompt, model, tool or retriever from quietly degrading it. The exam tests what to capture, the three grader types, and how evaluation fits into a delivery pipeline.

What to log

For every API call, capture enough to reproduce and attribute it:

Field	Why
request-id header	Support and tracing
Model and prompt version	Attribute behaviour to a specific configuration
usage (input, output, cache read, cache write)	Cost and cache health
stop_reason (and stop_details on refusals)	Truncation, refusals and tool-use rates
Latency, including time to first token when streaming	User experience and capacity planning
Tool calls made and their outcomes	Agent behaviour analysis
A user or session identifier	Abuse investigation and per-tenant cost; the request metadata.user_id field is designed for this and must not contain personal data

Store full prompts and responses where privacy policy allows; sampling a percentage is a common compromise. The Admin API exposes organisation-level usage and cost reports for finance. Claude Code can emit OpenTelemetry metrics and events (enabled through environment variables) covering sessions, tokens, cost, tool usage and permission decisions, which is how teams monitor adoption and spend across engineers.

Evaluation sets

An evaluation set is a collection of inputs with a way of judging outputs. Build it from real traffic where possible, add edge cases and known failures, and keep it versioned alongside the prompt. A few dozen cases will catch gross regressions; hundreds are needed for confident comparisons between variants. Keep a held-out portion that you never tune against, or your scores will flatter the prompt you optimised.

For agents, decide what you are grading. Outcome grading asks whether the final state is correct (the tests pass, the ticket was filed). Trajectory grading asks whether the path was acceptable (no forbidden tools called, no more than N turns, the right source consulted). Production agents usually need both: a correct result reached by deleting the test suite is not a success.

Graders

Grader	How it works	Strengths	Weaknesses
Code-based	Exact match, regex, schema validation, running tests, numeric tolerance	Cheap, fast, deterministic	Only for outputs with a checkable form
Model-based (LLM-as-judge)	A second model scores the output against a rubric or compares two outputs	Handles open-ended text; scales	Needs a precise rubric; must itself be validated against humans
Human	Experts rate samples	Ground truth for quality and safety	Slow and expensive; use to calibrate the others

Design model graders carefully: give them the rubric and the reference answer, ask for reasoning before a score, prefer binary or small-scale judgements to fine-grained ones, and periodically check their agreement with human raters. A grader that is never validated is a prompt with a number attached.

Regression testing

Treat the evaluation set as the test suite for your prompt. Run it automatically when any of these change: the prompt text, the model or its version, tool definitions, the retrieval pipeline, or the system prompt of a dependent agent. Compare against the previous baseline and block deployment if a metric falls below its threshold. Because model output has variance, run each case several times or use enough cases that the aggregate is stable, and look at per-case diffs rather than only the headline score.

# CI step sketch
- run: python evals/run.py --prompt prompts/support_v4.txt --model "$PINNED_MODEL" --out results.json
- run: python evals/gate.py results.json --min-accuracy 0.93 --max-refusals 0.02

Model upgrades deserve special attention: run the full set on the new model before switching, expect prompt adjustments, and keep the old model configured until the new one passes.

Closing the loop

Production monitoring feeds evaluation. Track rates over time (refusals, max_tokens truncations, tool errors, user corrections, thumbs-down) and sample failures into the evaluation set so that the next iteration targets what actually goes wrong. Alert on cache read rates dropping (a sign of an accidental prefix change) and on cost per request rising (a sign of context growth). Observability without evaluation tells you something changed; evaluation without observability tells you about yesterday's problems.

Exam focus

• Log request ID, model, prompt version, usage, stop_reason, latency and tool outcomes for every call; metadata.user_id identifies users without personal data.
• Code graders are deterministic but limited; model graders need validated rubrics; human grading calibrates both.
• Agent evaluation combines outcome grading (was the result right) with trajectory grading (was the path acceptable).
• Hold out part of the evaluation set to avoid tuning to the test.
• Run the evaluation set on every change to prompt, model, tools or retrieval, and gate deployment on thresholds.
• Monitor cache read rate, truncation rate and refusal rate in production and feed failures back into the evaluation set.