Evaluation, Testing & Optimisation

Build evals, test agents, run regressions and optimise cost and quality in production.

1. 4.1Designing EvaluationsHow to build an evaluation that can actually decide a question, from sourcing representative cases and choosing metrics to handling non-determinism and keeping the eval honest over time.10 min
2. 4.2Graders: Code, Model and HumanThe three families of graders, where each is trustworthy, how to design and calibrate a model grader, and how to combine them so that cost, speed and judgement are balanced.9 min
3. 4.3Testing Agents and Tool UseWhy agents need a different test strategy from single calls, the levels of testing from tool unit tests to end-to-end trajectory evaluation, and how to control side effects and non-determinism.10 min
4. 4.4Regression Suites and CIHow to make prompts, tools and model configuration first-class, tested artefacts in a delivery pipeline, with gates that catch regressions without making the pipeline too slow or expensive to run.9 min
5. 4.5Optimising Cost and LatencyA practical method for reducing what a Claude system costs and how long it takes, from profiling tokens by route to applying levers in a safe order and proving each change against evaluations.10 min
6. 4.6Production Monitoring and Feedback LoopsWhat to observe in a live Claude system across quality, safety, reliability and cost, how to instrument it, and how to turn production signals back into evaluation cases and better prompts.9 min