Lesson 6.3
Explaining Capabilities and Limits
How to give executives, product owners and users an accurate mental model of what Claude systems can and cannot do, how to talk about accuracy and risk, and how to demo and document without over-promising.
Most disappointment with AI projects is a gap between what stakeholders imagined and what the system does. Closing that gap is the architect's job, and it requires translation: from model behaviour to business language, without either hype or hedging. The professional exam tests whether you can explain the right things in the right words to the right audience.
The mental model to install
A few ideas, stated plainly, prevent most misunderstandings:
- It is probabilistic. The same input can produce different outputs; "it worked in the demo" is one sample. Reliability is measured as a rate over many runs.
- Capability is not reliability. A model that can do a task brilliantly sometimes is a different thing from a system that does it acceptably every time. Production engineering is about the second.
- It can be confidently wrong. Fluent output is not evidence of correctness. Grounding, citations and validation exist because of this.
- It knows what it was given. No persistent memory unless you build it; no knowledge of events after its training data unless you provide them through tools or documents.
- Tools change what it can do, and what it can get wrong. Giving it the ability to act is the moment risk changes character.
- Context is finite. It cannot read the whole archive at once; what you put in front of it determines what it can use.
An analogy that works with many audiences: a very capable new colleague who has read a great deal, works fast, has no memory between meetings unless you take notes for them, and must have their work checked in proportion to its consequences.
Translating stakeholder questions
| What they ask | What they usually mean | What to answer |
|---|---|---|
| "How accurate is it?" | Can I trust it? | Give the eval rate by segment and the standard used (every time versus best of several); describe what happens on failure |
| "Can it just do X as well?" | Scope creep, or a real need | Explain the ladder of autonomy and what adding X costs in risk and evaluation |
| "Why did it say that?" | Is it reliable? | Explain that explanations are reconstructions; show the evidence and controls instead |
| "Will it replace the team?" | Fear or budget planning | Describe which tasks it takes, assists and leaves; show the oversight design |
| "Is our data safe?" | Compliance and reputation | Walk through the data flow, retention and agreements accurately |
| "When will it be done?" | Timeline | Separate build from evaluation and from rollout; explain why quality takes iteration |
Talking about accuracy honestly
Never quote a single accuracy number without its conditions: which dataset, which segments, how many runs, which standard. Explain pass^k in plain terms ("how often it gets all ten right") when the system acts without review. Show the failure examples alongside the successes; stakeholders who have seen a realistic failure design better processes around it and trust the architect more. Distinguish errors the system flags from errors it does not; the second kind drives the oversight design.
Demoing responsibly
Demos set expectations harder than any document. Demo the real system on real-shaped data, include a case where it declines or asks for help, show a wrong answer and how the controls catch it, and show the cost and latency as they happen. Avoid rehearsed inputs that always work; if the audience suggests an input, take it. A demo that fails gracefully in front of the sponsor is worth more than one that succeeds suspiciously.
Communicating risk
Use the risk register. Say what the system cannot be prevented from doing (being wrong sometimes; being targeted by injection), what the architecture ensures nonetheless (no consequential action without review; no exfiltration path), and what residual risk the organisation is accepting. Executives are used to accepting residual risk; they are not used to being told there is none and then finding out.
Documentation and language
Write user-facing and internal documentation in terms of behaviour, not personality. "The assistant suggests; a reviewer approves" rather than "the AI decides". Avoid anthropomorphic language that implies intent or understanding; it inflates expectations and complicates accountability. State limits explicitly in the places people will see them, including in the interface.
Managing expectations over time
Capabilities improve with each model generation and each prompt iteration, which creates a different risk: stakeholders expect the trajectory to continue on demand. Tie expectation setting to the roadmap and to measured evidence: what will be evaluated next, what would have to be true to extend autonomy, and what the model lifecycle means for stability. Report actuals against the business case at the agreed checkpoints, including the misses.
Exam focus
- Install the core mental model: probabilistic, capable but not uniformly reliable, confidently wrong at times, limited to what it is given, changed by tools, bounded by context.
- Quote accuracy only with dataset, segments, repeat runs and the standard (pass@k versus pass^k); show failures alongside successes.
- Translate stakeholder questions into the underlying concern and answer with evidence and controls, not reassurance.
- Demo real behaviour including declines, failures and the controls that catch them.
- Communicate residual risk explicitly using the risk register.
- Use behavioural, non-anthropomorphic language in documentation and interfaces.
Check yourself
5 questions
1.An executive sponsor asks why the assistant 'gave two different answers to the same question' and whether the system is broken. Which explanation is MOST accurate and useful?
2.A product manager expects the assistant to answer questions about events that happened last week and about the company's internal data, with no additional integration. Which explanation should the architect give?
3.A vendor demo of an agent impressed the leadership team: it completed a complex task flawlessly. Leadership now expects the same reliability across all tasks from day one. What should the architect communicate?
4.A compliance director insists that the assistant must be '100% accurate or we cannot use it'. The use case is first-pass document review before human sign-off. How should the architect respond?
5.A business stakeholder says: 'Every time a user corrects the assistant, it should learn so it never makes that mistake again.' Which response sets expectations CORRECTLY?