Optimising Cost and Latency

The trade-offs lesson in the design domain explained the levers. This lesson is the operating procedure: how to find where the money and the seconds go, which lever to pull first, and how to prove that each change kept quality intact. It assumes an eval set exists; without one, optimisation is guessing with a dashboard.

Step 1: Profile before you touch anything

Pull token usage per route from your logs or from the usage and cost reporting available through the Admin API. Break it down into:

• Uncached input tokens, cached reads and cache writes.
• Output tokens, including thinking.
• Tool and server-tool usage fees.
• Requests per task and turns per agent run.

Rank routes by spend. Optimisation effort follows the ranking; a 40% saving on a route that is 2% of the bill is not worth a sprint. For latency, pull time to first token and total duration percentiles per route and identify whether the time is in model generation, tool calls or your own pipeline.

Step 2: Free wins

These rarely cost quality and should be exhausted first.

Lever	Typical symptom	Action	Verify
Caching	Low or zero cache reads on repeated traffic	Stabilise the prefix, fix tool order, move volatile content last, add breakpoints	Cache hit rate rises; eval unchanged
Input hygiene	Bloated prompts, stale tool results, oversized retrieval	Prune instructions, cap tool result size, tighten top-k, clear consumed results	Input tokens fall; eval unchanged or better
Output hygiene	Verbose preambles, repeated content, unneeded explanations	Specify format and length; ask for the answer only	Output tokens fall; eval unchanged
Loop hygiene	Agents taking many turns, redundant tool calls	Better tool descriptions, parallel tool calls, clearer stopping criteria	Turns per task fall
Batching	Offline work on the synchronous path	Move to the Message Batches API	Cost falls; latency irrelevant
max_tokens discipline	Truncation retries, or oversized caps on short routes	Set per route to the realistic maximum	Fewer retries

Step 3: Trades, measured

Only after the free wins, in this order:

1. Effort. Lower the effort setting on routine routes and confirm against the eval. On recent models this is often the largest safe saving and preserves a single cache namespace.
2. Model tier. Try the next tier down per route. Reject if any segment fails.
3. Routing. Place a cheap classifier in front to send easy traffic to a cheaper configuration. Evaluate the router separately; its mistakes are quality incidents.
4. Thinking posture. For routes where reasoning adds nothing, reduce it through effort; where it adds accuracy, keep it and pay.

Each trade is a reviewed change with an eval result attached. Judge by cost per completed task.

Latency specifically

• Stream everything interactive; time to first token is what users feel.
• Parallelise independent tool calls in one turn and return all results in one message.
• Reduce turns: better tool descriptions and richer tool results mean fewer round trips.
• Prefetch data you know the model will need and include it in the prompt rather than waiting for a tool call.
• Shorten outputs; generation time is linear in output tokens.
• Move work off the path: pre-compute summaries, cache retrieval, do background enrichment.
• Capacity options: where the platform offers priority or faster service tiers, they trade money for predictability; confirm availability and use for the routes that justify it.
• Regional and platform choices affect network latency; measure from where your users are.

Attribution and budgets

Give each team, product or tenant its own workspace or metadata tag so that cost is attributable. Set budget alerts per workspace. Report cost per completed task and cost per active user alongside raw spend; these are the numbers finance and product can act on. Make cache hit rate a first-class metric with alerts, because a cache regression is the most common cause of a sudden cost spike.

Worked sequence

A document QA route is the top line item. Profiling shows 70% of tokens are uncached input: the policy corpus is resent every call with a per-request header line. Moving the header after the breakpoint turns most of that into cache reads. Output hygiene removes a habitual two-paragraph preamble. A lower effort setting passes the eval on all segments except legal queries, so a small router sends legal queries to the higher setting. The next tier down fails the multilingual segment and is rejected. Cost per completed task falls by roughly two thirds with no measured quality loss; each step is documented with its eval run.

Anti-patterns

• Switching to a smaller model first because it is the obvious lever.
• Measuring cost per request while turns per task rise.
• Optimising a prompt on a route that is not in the top spenders.
• Accepting a quality regression because it is "within noise" without repeated runs.
• Cost savings without cost attribution, so nobody knows which team benefited.

Exam focus

• Profile first: tokens by type, tool fees, turns per task, by route; prioritise by spend.
• Exhaust free wins (caching, input and output hygiene, loop hygiene, batching, max_tokens discipline) before trading quality.
• Trade in order: effort, then model tier, then routing, each proven against evals and judged by cost per completed task.
• Latency levers: streaming, parallel tool calls, fewer turns, prefetching, shorter outputs, work off the critical path.
• Attribute cost by workspace or metadata; alert on cache hit rate and budgets.
• The most common anti-pattern is reaching for a smaller model before fixing caching and hygiene.