Evals, Testing and Debugging Claude Features

A prompt that "seems to work" is not a tested feature. Evaluation turns subjective judgement into numbers you can track across prompt edits and model upgrades, and disciplined debugging uses the response fields and headers the API returns rather than guesswork. The exam asks you to pick the right grading method for a scenario and to diagnose common failures from their symptoms.

Build the eval set first

An eval is a set of inputs with expected outcomes plus a way to score each result. Start small and representative:

• Pull real inputs from logs or transcripts (anonymised), not invented ones.
• Include edge cases: empty input, very long input, adversarial phrasing, non-English text.
• Record an expected result or a rubric for each case.
• Version the set alongside the prompt it tests.

Even 30 well-chosen cases catch most regressions. Grow the set whenever production surfaces a new failure.

Three kinds of grader

Grader	Use when	Example
Code-based	Output is deterministic or checkable	Exact match, regex, JSON schema validation, unit tests passing
Model-based	Quality is subjective but describable	Claude scores a summary against a rubric and returns a 1-5 rating
Human	Stakes are high or the rubric is still forming	Reviewers label a sample each week

Prefer code-based grading wherever possible: it is cheap, fast and reproducible. Model-based grading works well when you give the grader a precise rubric, ask for a structured verdict, and spot-check it against human labels. Never let the model grade its own free-text answer without a rubric; "does this look good?" produces inflated scores.

A minimal eval harness

import json, anthropic
client = anthropic.Anthropic()

def run_case(case):
    resp = client.messages.create(
        model=MODEL, max_tokens=300, system=SYSTEM_PROMPT,
        messages=[{"role": "user", "content": case["input"]}],
    )
    text = next(b.text for b in resp.content if b.type == "text")
    return {"id": case["id"], "output": text,
            "pass": grade(case, text), "usage": resp.usage.model_dump()}

results = [run_case(c) for c in json.load(open("evals/tickets.json"))]
score = sum(r["pass"] for r in results) / len(results)
print(f"pass rate {score:.0%}")

Run it in CI on every prompt change and fail the build when the pass rate drops below a threshold. Because model outputs vary, run each case more than once if your budget allows and compare distributions rather than single runs. For bulk evals, the Message Batches API cuts cost substantially.

Testing without hitting the API

Unit tests for your orchestration code should not spend tokens or depend on network. Mock the client and return canned Message objects, including ones with stop_reason: "tool_use" and stop_reason: "max_tokens", so the loop logic around the API is exercised. Keep a separate, smaller suite of integration tests that call the real API against a pinned model.

Debugging with what the API gives you

Every response carries id, and every HTTP response carries a request-id header. Log both with your own correlation ID; Anthropic support needs the request ID to investigate. Log usage and stop_reason on every call so you can see cost spikes and truncation rates in a dashboard.

Symptom	Likely cause	Check
Output ends mid-sentence	max_tokens too low	stop_reason == "max_tokens"
Tool never called	Weak tool description or tool_choice: none	Inspect content for tool_use; improve description
400 about roles alternating	Appended two user turns, or history started with assistant	Print messages roles before the call
400 about tool_use_id	Dropped or reordered tool results	Each tool_result must follow the assistant turn it answers
Cache never hits	Something volatile in the prefix	usage.cache_read_input_tokens stays 0
Sudden 404	Model ID typo or deprecated model	Compare with client.models.list()
Intermittent 429 / 529	Rate limit or overload	Read retry-after; rely on SDK retries

Most "the model is being weird" reports turn out to be one of these.

Prompt changes are code changes

Keep prompts in version control, give them identifiers, and record which prompt version and model produced each logged response. When you upgrade a model, re-run the full eval before switching production traffic; behaviour and token counts both shift between generations. Treat a drop in pass rate exactly like a failing test.

Observability in production

Sample a percentage of real traffic into a review queue, track pass rate from your model-based grader over time, and alert on drift in output length, refusal rate and max_tokens frequency. These signals usually move before users complain.

Exam focus

• An eval is a versioned set of representative inputs plus a grader; build it from real data and grow it from production failures.
• Code-based graders are preferred when outputs are checkable; model-based graders need an explicit rubric and spot-checking; human review is for high stakes.
• Mock the client in unit tests to exercise loop logic, including tool_use and max_tokens stop reasons, without spending tokens.
• Log the response id or request-id header plus usage and stop_reason on every call.
• Truncated output means stop_reason: "max_tokens"; a zero cache_read_input_tokens means the prefix is changing.
• Re-run the eval before any prompt or model change reaches production.