Designing Evaluations

Every other decision in this certification, from model choice to prompt wording to effort settings, is supposed to be made "against the evals". That presumes the evals exist and can be trusted. Building one is a design task in its own right, and the professional exam treats a weak eval as a worse outcome than no eval, because it manufactures false confidence.

Anatomy of an eval

Component	What it is	Common failure
Dataset	Inputs with expected outputs, references or rubrics	Unrepresentative, too small, stale
Task runner	Code that executes the system under test on each case	Tests a prompt in isolation rather than the real pipeline
Grader	Decides whether an output is acceptable	Measures the wrong thing; see the graders lesson
Metrics	Aggregates grader results	A single average hides the failing segment
Threshold	The bar for a pass	Set after seeing results, so it always passes

Run the eval against the system as deployed: the same prompt assembly, retrieval, tools and model configuration. An eval of the prompt alone will not catch a retrieval regression.

Sourcing cases

Representativeness beats volume. Draw from:

• Production traffic, sampled across routes, tenants, languages and time, with sensitive data handled under the same rules as logs.
• Known failures, every incident becomes a case.
• Edge cases and adversarial inputs: empty documents, contradictory instructions, injection attempts, inputs at the context limit.
• Synthetic cases generated by a model to fill gaps, reviewed by humans before inclusion, and labelled as synthetic so their share can be tracked.

Label quality matters more than label quantity. Two reviewers on a sample, with disagreement measured and resolved, is the minimum for cases that will gate releases.

Choosing metrics

Match the metric to the decision:

• Accuracy or pass rate for classification and extraction.
• Field-level precision and recall for structured extraction, so one wrong field does not hide behind nine right ones.
• Rubric score for open-ended generation, graded item by item.
• Task success for agents: did the end state meet the specification?
• Operational metrics alongside quality: tokens, cost per case, latency, number of turns.

Report by segment, not only overall. A 94% average with 60% on one document type is a failing system for that document type.

Non-determinism

Model outputs vary between runs even at identical settings. A single run can flip a borderline case and masquerade as a regression or an improvement. Run each case several times and report the distribution. Two useful aggregates: pass@k, the chance that at least one of k attempts passes, which matters when a human picks the best of several drafts, and pass^k, the chance that all k attempts pass, which matters when the system must be right every time with no human filter. Production systems without a human in the loop should be judged on pass^k.

Keeping the eval honest

• Hold out a test split. If you tune prompts against the whole set, you overfit to it. Iterate against a training split, check a validation split, and report the test split only.
• Watch for saturation. When everything passes, the eval has stopped discriminating. Add harder cases or retire it as a smoke test.
• Watch for contamination. Cases copied into the prompt as examples are no longer tests.
• Version the dataset. Every change to cases or labels is a change to the yardstick; record it alongside results.
• Audit periodically. Re-sample production to check the eval still reflects reality.

Cost and cadence

Evals cost tokens. Use the Message Batches API for full runs; it halves the cost and parallelises. Share cached prefixes where cases use the same system prompt. Keep a small fast suite for every change and a full suite for releases and model migrations. Track eval spend as a line item; an eval nobody can afford to run will not be run.

A minimal case format

{
  "id": "claims-0412",
  "segment": "handwritten-form",
  "input": {"document_file_id": "file_...", "instructions": "..."},
  "expected": {"claim_amount": 1240.50, "currency": "EUR"},
  "grader": "field_match",
  "source": "production-2026-02",
  "notes": "Amount written with comma decimal separator"
}

Fields for segment, source and notes are what make the dataset maintainable a year later.

Exam focus

• An eval has a dataset, runner, grader, metrics and threshold; it must exercise the deployed pipeline, not an isolated prompt.
• Source cases from production, incidents, adversarial inputs and reviewed synthetic data; label quality outranks quantity.
• Report metrics by segment; use field-level measures for extraction and task success for agents.
• Handle non-determinism with repeated runs; pass^k is the right standard for unsupervised production systems.
• Hold out a test split, watch for saturation and contamination, and version the dataset.
• Run full suites through Batches to control cost; keep a fast subset for every change.