Lesson 4.4
Regression Suites and CI
How to make prompts, tools and model configuration first-class, tested artefacts in a delivery pipeline, with gates that catch regressions without making the pipeline too slow or expensive to run.
A prompt edit is a code change. A model version bump is a dependency upgrade. A new tool description is an API change. Each can regress behaviour silently, and none is caught by conventional tests. The professional architect wires evaluation into continuous integration so that the question "did this make things worse?" is answered automatically before production finds out.
What triggers a run
| Change | Risk | Minimum suite |
|---|---|---|
| Prompt or template edit | Behaviour shift on affected route | Fast suite for the route; full suite before release |
| Tool schema or description change | Tool-selection regressions | Tool-selection tests plus affected trajectories |
| Model version or effort change | Broad behaviour shift, tokenizer changes | Full suite, repeated runs, cost and latency comparison |
| Retrieval or index change | Grounding regressions | Grounding and citation checks |
| Guardrail or policy change | Over- or under-refusal | Safety suite plus false-refusal cases |
| Dependency or SDK upgrade | Request shape differences | Contract tests against recorded requests |
Treat all of these as changes to a versioned bundle: prompt version, model ID, tool definitions, retrieval configuration, effort and thinking settings. The bundle is what you test and what you deploy.
Suite tiers
Running every case on every commit is unaffordable and slow. Tier the suites:
- Smoke (seconds to a minute): schema checks, a handful of canonical cases per route, tool-selection tests. Runs on every commit.
- Route suite (minutes): the full dataset for the routes touched by the change. Runs on pull requests.
- Full regression (tens of minutes to hours, often via Batches): all routes, repeated runs for pass^k, cost and latency baselines. Runs nightly and on release candidates.
- Migration suite: full regression plus comparison reports between two model versions. Runs when a model changes.
Gates and thresholds
A gate compares the candidate against the current baseline, not against an absolute number alone. Fail the build when a segment drops more than an agreed tolerance, when any safety case fails, when cost per case rises above budget, or when latency percentiles regress. Record the thresholds in the repository alongside the suite so that loosening a gate is itself a reviewed change.
gates:
overall_pass_rate: {min: 0.92, max_drop: 0.02}
segment_pass_rate: {max_drop: 0.05}
safety_cases: {must_pass: all}
cost_per_case_usd: {max_increase_pct: 15}
p95_latency_ms: {max_increase_pct: 20}
Flakiness
Non-deterministic outputs make naive gates flaky, and flaky gates get ignored. Mitigate by running borderline cases several times and gating on the aggregate, by using confidence intervals rather than point comparisons for small datasets, and by setting a flakiness budget: a known set of volatile cases that inform but do not block. Investigate when a case joins the volatile set; volatility is often a sign that the prompt is ambiguous for that input.
Cost control
Use the Message Batches API for the full and migration tiers. Cache shared prefixes, which also keeps CI requests representative of production caching. Sample large datasets for the route tier with stratification by segment. Budget eval spend per month and report it; it is part of the cost of the system.
Baselines and golden datasets
Keep the dataset versioned in the repository or an artefact store, with labels reviewed through pull requests. Store baseline results per bundle version so that comparisons are reproducible. When labels change, re-run the baseline before comparing; otherwise you are measuring the label change.
Shadow runs and canaries
CI tells you about known cases. Production tells you about unknown ones. Before switching a bundle, run it in shadow against live traffic, grading the outputs without serving them, and compare to the current bundle. Then canary to a small share of traffic with monitoring on quality and cost signals, and promote gradually. The regression suite and the shadow run answer different questions; use both for model migrations.
Claude Code and the Agent SDK in CI
Headless Claude Code and the Agent SDK can run inside pipelines: generating test cases, triaging failures, reviewing diffs. When they do, they are production agents and need the same controls: constrained tools, read-only by default, scoped credentials, turn and budget limits, and their own tests. The output of an agent in CI should be a proposal (a comment, a draft pull request), with humans or deterministic checks deciding merges.
Rollback
A regression that reaches production must be reversible by configuration. Because the bundle is versioned and the gateway pins versions per route, rollback is a pointer change, not a redeploy. Keep the previous bundle warm in caches where traffic allows.
Exam focus
- Prompts, tools, model IDs and settings form a versioned bundle; any change to it triggers evaluation.
- Tier suites: smoke on every commit, route suite on pull requests, full regression via Batches nightly and on releases, migration suite on model changes.
- Gates compare against baselines by segment, include safety must-pass cases, and cover cost and latency, with thresholds under version control.
- Manage flakiness with repeated runs, confidence intervals and a declared flakiness budget.
- Shadow runs and canaries complement CI by exercising unknown production inputs.
- Agents used inside CI are production agents and need the same constraints; their output should be proposals, not merges.
Check yourself
5 questions
1.A team has a good evaluation suite but runs it manually about once a month. Regressions are often discovered by customers in between. Which process change is MOST effective?
2.A regression suite has grown to 4,000 items and now costs a significant amount and takes two hours per run, so developers have started skipping it. Which strategy preserves protection while making the suite practical?
3.A CI regression gate fails about once in every five runs on unchanged code, and developers have begun re-running until it passes. Which approach addresses the root cause rather than the symptom?
4.A regression suite that had been stable began failing on several items overnight, although nothing in the repository changed. The configuration specifies the model with a floating alias intended to always resolve to the newest version. What should the architect recommend?
5.A prompt change passes the offline regression suite, but the team is nervous because the suite cannot cover the full diversity of production traffic. Which release practice MOST reduces the risk of an undetected regression reaching all users?