From Pilot to Production

Pilots succeed and production launches fail for predictable reasons: the pilot ran on curated data, nobody owned the system afterwards, the controls that make production safe were deferred, and the rollout was a switch rather than a ramp. The professional exam expects you to design the path from pilot to production as deliberately as the architecture itself.

Designing a pilot that transfers

Pilot property	Why it matters
Real data under real governance	Results on cleaned samples do not predict production; privacy approvals take time, so start them before the pilot
Real users doing real work	Adoption and workflow fit are measured, not assumed
Narrow scope with explicit exclusions	A pilot that tries to do everything proves nothing
Pre-registered success criteria	Quality thresholds, cost per task, time saved and adoption targets written before the pilot starts
An evaluation set built during the pilot	The pilot's labelled cases become the regression suite
Production-like architecture	Gateway, logging, permissions and tool scoping in place, even at small scale
A time box and a decision rule	Scale, iterate or stop; the pilot ends on the date regardless

Common traps: optimising for the demo rather than the metric; letting the pilot team be the only users, so feedback is friendly; leaving evals and monitoring "for production"; and allowing scope creep that leaves no clean measurement.

Production readiness criteria

Before promotion, each of these should be demonstrably in place:

1. Quality: eval suite passes thresholds by segment; pass^k measured for unsupervised actions.
2. Security and safety: threat model and risk register; injection defences tested; least-privilege tools; approval gates on consequential actions.
3. Privacy and compliance: data flows mapped; retention settings confirmed; agreements signed; impact assessment filed; user disclosure ready.
4. Reliability: timeouts, retries, circuit breakers, idempotency; rate-limit headroom; workspace isolation; load tested.
5. Cost controls: caching verified; budgets and alerts per workspace; turn and spend limits on agents.
6. Observability: tracing, dashboards, sampled production grading, alerting on the four signal families.
7. Operations: runbooks for AI-specific incidents; on-call ownership; rollback by configuration; change control for prompts and models.
8. People: users trained; support channel; feedback mechanism; reviewers staffed for oversight volumes.
9. Documentation: architecture decision records, prompt and tool catalogues, handover complete.

Hold a readiness review against this list with security, privacy, operations and the business owner present. Items can be accepted as risks, but explicitly and with owners.

Staged rollout

shadow   -> system runs on live traffic, outputs graded, nothing served
canary   -> small share of users or cases; compare metrics to control
ramp     -> increase share in steps, holding at each until metrics are stable
full     -> all traffic; pilot team hands over to the operating team

At each stage, define the metrics that gate progression and the conditions that trigger rollback. Keep a control group as long as you can; it is the only clean measure of impact. For agentic systems, ramp oversight down as well as traffic up: start in-the-loop, move to on-the-loop when override rates and evals justify it.

Operating model

Production systems need owners for things pilots do not:

Responsibility	Typical owner
Prompt and tool changes, with eval gates	Product engineering team
Evaluation datasets, graders, calibration	Quality or ML engineering
Model version and migration decisions	Platform or architecture team
Cost budgets and attribution	Platform team with finance
Human review queues and SLAs	Business operations
Security, privacy and policy reviews	Risk functions, on a defined cadence
Incident response	On-call rotation with runbooks

Write the model down and get it agreed before launch. Ambiguity here is the most common cause of post-launch decay.

Change management

Users and affected staff need to know what is changing, why, what the system can and cannot do, how to give feedback and how to escalate. Train on the realistic behaviour, including failure modes, not on the demo. Where roles change, involve the people early; a system that is quietly resisted will be quietly abandoned. Tie communication to the business case so that the benefits being claimed are the ones people experience.

Scaling considerations

Production volume exposes limits the pilot never touched: rate limits per workspace, downstream systems not sized for agent traffic, reviewer capacity, and cost at scale. Model the ramp in advance, request limit increases early, and load test the whole path including tools. Plan capacity for peak, not average, and know which capacity options your platform offers.

Exam focus

• A transferable pilot uses real data and users, narrow scope, pre-registered criteria, a production-like architecture and a time box with a decision rule.
• Readiness spans quality, security, privacy, reliability, cost, observability, operations, people and documentation; review it formally with risks accepted explicitly.
• Roll out in stages (shadow, canary, ramp, full) with gating metrics, rollback conditions and a control group.
• Ramp oversight down only as evidence accumulates; do not switch from in-the-loop to autonomous in one step.
• An agreed operating model with named owners prevents post-launch decay.
• Scale planning covers rate limits, downstream systems, reviewer capacity and peak cost.