Lesson 5.1
Threat Modelling Agentic Systems
How to threat model a Claude system that reads untrusted content and acts through tools, the attack vectors specific to agents, and how to size and shrink the blast radius before an attacker finds it.
Classical threat modelling asks what an attacker can reach and what they can do once there. Agentic systems add a new kind of component to the diagram: one that reads text and may act on it. Every document, web page, tool result and memory entry becomes a potential instruction channel, and every tool becomes a potential action an attacker can trigger. The professional exam expects you to draw the boundaries, enumerate the vectors and design the blast radius down.
Draw the trust boundaries
Start with a data-flow diagram and mark who controls each input:
| Source | Controlled by | Trust level |
|---|---|---|
| System prompt and operator messages | You | Trusted |
| End-user turns | The user | Semi-trusted; the user may be the attacker |
| Retrieved documents | Whoever wrote them | Untrusted |
| Tool and MCP results | External systems and their data | Untrusted |
| Web search and fetch results | The internet | Untrusted |
| Agent memory | The agent's past sessions, possibly poisoned | Untrusted on read |
| Tool and MCP server definitions | Server authors | Trusted only after review and pinning |
The model sits inside the boundary, consuming all of these, and its outputs drive tools that cross back out. The key insight is that the model cannot reliably distinguish instructions from data on its own; your architecture must.
Agent-specific attack vectors
- Direct prompt injection. The user tries to override operator instructions. Mostly a policy and robustness issue; dangerous when it unlocks tools.
- Indirect prompt injection. Instructions hidden in content the agent reads: a web page, a PDF, an email, a database field, a tool description. The attacker never talks to the model; they plant text where the model will read it.
- Data exfiltration through tools. The agent is induced to encode private data into a URL it fetches, an email it sends, a file it writes to a shared location, or a search it performs.
- Privilege escalation through over-scoped tools. A read-only assistant with a "run SQL" tool is a write-capable assistant.
- Tool poisoning and rug pulls. An MCP server's tool descriptions contain instructions, or a trusted server changes behaviour after approval.
- Memory poisoning. Injected content persuades the agent to write a malicious instruction into long-term memory, where it executes in every future session.
- Denial of wallet. Inputs that cause runaway loops, enormous tool results or repeated expensive calls.
- Approval fatigue. A flood of benign confirmation requests trains the human to approve the malicious one.
- Supply chain. Skills, plugins, MCP servers and prompts pulled from registries without review.
The lethal trifecta
A useful heuristic: an agent is dangerous when it combines three capabilities at once: access to private data, exposure to untrusted content, and a channel to communicate externally (send, post, fetch arbitrary URLs). Any two are manageable; all three mean a successful injection can exfiltrate. Design so that no single agent, or no single session, holds all three. Split the work: a reader agent with private data and no external channel hands structured results to a sender agent that never sees untrusted content.
Sizing the blast radius
For each agent, write down the worst thing it could do if fully compromised: every tool it can call, with the broadest arguments, under its credentials, for the duration of a session. That is the blast radius. Shrink it:
| Lever | Effect |
|---|---|
| Least-privilege tools | Narrow operations, narrow arguments, read before write |
| Separate read and write agents | Breaks the trifecta |
| User-delegated credentials | Agent can do nothing the user could not |
| Confirmation on consequential actions | Human sees the action before it happens |
| Egress allowlists | No arbitrary URLs, no arbitrary recipients |
| Session and spend limits | Caps denial of wallet |
| Disposable sandboxes | Compromise does not persist |
| Memory write review | Poisoning does not persist |
Build a risk register
Threat modelling produces a register: each threat, its vector, likelihood, impact, the controls in place, the residual risk and an owner. Keep it with the architecture decision records, review it at every change to the tool set or data sources, and use it to answer the security team's questions before they ask. The register is also what makes the human-oversight design defensible: it shows why a given action needs a confirmation step.
Test the model you drew
A threat model that is never exercised is a drawing. Turn each vector into adversarial cases in the regression suite: a document that instructs the agent to email its context, a tool result that claims administrator authority, a web page with a URL template for exfiltration. The agent must refuse and should flag. Re-run on every prompt, tool and model change.
Exam focus
- Mark every input by who controls it; retrieved content, tool results, web content and memory are untrusted.
- Know the agent-specific vectors: indirect injection, exfiltration through tools, over-scoped tools, tool poisoning, memory poisoning, denial of wallet, approval fatigue, supply chain.
- The lethal trifecta is private data plus untrusted content plus an external channel; split agents so no one holds all three.
- Blast radius is the worst a fully compromised agent could do; shrink it with least privilege, read/write separation, user credentials, confirmations, egress allowlists and limits.
- Maintain a risk register with owners and review it at every tool or data change.
- Convert each threat into standing adversarial test cases.
Check yourself
5 questions
1.An executive assistant agent can read the executive's email and calendar, browse arbitrary web pages to research topics, and send emails on the executive's behalf. The security team asks the architect to assess the design. Which assessment is MOST accurate?
2.During a threat-modelling workshop for a retrieval-augmented support agent, a developer argues that the system prompt is a trust boundary: 'if we tell the model what it may do, it will only do that'. Which statement BEST corrects this view?
3.A customer-support agent currently has a single 'crm' tool capable of reading, updating and deleting any record. Most conversations only need to look up order status. Which change MOST reduces the agent's risk profile?
4.A threat model identifies that an internal coding agent, if compromised, could use its credentials to modify any repository in the organisation. Which mitigation BEST limits the blast radius?
5.A product team asks when in the lifecycle they should threat-model their Claude agent. Which answer reflects good practice?