Documentation, Runbooks and Handover

Claude systems carry more tacit knowledge than conventional software: why a prompt is phrased as it is, which inputs the model struggles with, what the eval thresholds mean, which tool was scoped narrowly after an incident. If that knowledge lives only in the heads of the build team, the system decays the day they move on. The professional architect leaves behind documentation that lets someone else operate, change and defend the system.

The documentation set

Artefact	Contents	Owner after handover
Architecture overview	Data-flow diagram with trust boundaries; component list; platform and region choices	Architecture or platform team
Architecture decision records	Each significant choice with context, options, decision and consequences	Architecture team; appended by change owners
Risk register	Threats, controls, residual risk, owners, review dates	Risk function with engineering
Prompt catalogue	Every prompt and template with version, model validated on, eval score, rationale per section, known failure modes	Product engineering
Tool and MCP catalogue	Each tool: purpose, schema, permissions, credentials pattern, owner, rate limits, idempotency behaviour	Integration owners
Evaluation documentation	Datasets with provenance, graders and calibration results, thresholds and gate configuration, how to run	Quality engineering
Data and privacy records	Data-flow map, classification per route, retention settings, agreements, impact assessment	Privacy function
Operating runbooks	Procedures for incidents and routine operations	Operations with on-call
Change process	How prompts, tools, models and settings are changed, tested and released	Platform team
Known limitations and user guidance	What the system does not do; how users escalate	Product owner

Keep documentation next to the code and configuration it describes, under version control, and reviewed in the same pull requests. A separate wiki that nobody updates is where documentation goes to die.

Writing a prompt catalogue entry

A prompt without rationale is unmaintainable; the next engineer will not know which sentences are load-bearing. For each section of a prompt record why it exists, which eval cases it affects, and what happened when it was tried without. Record the model it was validated on, because behaviour shifts between generations and a migration will revisit every sentence.

## claims-extraction v2026-03-14.2
Model validated: <pinned id>   Eval: field_accuracy 0.94 (segment handwritten 0.88)
Section <instructions> para 3: "Report amounts with currency codes" — added after
  incident INC-221 (mixed currencies). Removing drops field_accuracy to 0.91.
Known failures: rotated scans; amounts with spaces as thousands separators.

Runbooks for AI-specific incidents

Conventional runbooks cover outages. Add the incidents that only AI systems have, each starting from the trace ID and the deployed bundle version:

• Quality regression after a change. Confirm with sampled grading; roll back the bundle; open a case review.
• Suspected prompt injection with action taken. Disable the affected tool; query sessions by trace for the same pattern; assess impact; notify; add adversarial cases.
• Cost runaway. Identify the route; check cache hit rate and turns per task; kill sessions; lower limits; fix and re-enable.
• Refusal spike. Compare against recent prompt, policy and model changes; sample refused cases; adjust context or escalate to the vendor.
• Rate limiting or overload. Confirm backoff behaviour; shed load by priority; request limit changes; communicate.
• Model deprecation notice. Trigger the migration process with its owner and timeline.
• Credential exposure. Rotate through the Admin API or console; audit usage; review logs for leakage path.
• Data subject request. Locate and delete personal data across logs, memory stores, eval datasets and files.

Each runbook names the owner, the decision authority for rollback and the communication template.

Handover

Treat handover as a project phase with acceptance criteria, not a meeting. Include:

1. Walkthroughs of architecture, prompts, evals and runbooks with the receiving team, recorded.
2. Shadowing, where the receiving team handles changes and incidents with the build team observing, then the reverse.
3. A dry run of the most likely incident and of a prompt change through the full change process.
4. Access and ownership transfer: workspaces, keys, repositories, dashboards, on-call rotations.
5. A known-issues list with the build team's honest assessment of fragility.
6. Acceptance sign-off against a checklist agreed in advance.

Living documentation

Documentation that is not maintained is misleading, which is worse than absent. Make updates part of the definition of done for every change; include documentation review in release gates; schedule quarterly reviews of the risk register and runbooks; and generate what can be generated (tool catalogues from schemas, eval reports from suite runs) so that the hand-maintained part stays small.

Exam focus

• The documentation set includes architecture, ADRs, risk register, prompt catalogue, tool catalogue, evaluation docs, privacy records, runbooks, change process and user guidance, each with an owner.
• Prompt catalogue entries record rationale per section, the model validated on, eval scores and known failures.
• AI-specific runbooks cover quality regressions, injection incidents, cost runaways, refusal spikes, deprecations, credential exposure and data subject requests, each starting from trace ID and bundle version.
• Handover is a phase with walkthroughs, shadowing, incident dry runs, access transfer and acceptance criteria.
• Keep documentation in version control beside the code and update it as part of every change.