Latency, Throughput and Rate Limits

Latency is what users feel; throughput is what your organisation is allowed; both are bounded by limits that return specific HTTP status codes when exceeded. The exam expects you to read those codes correctly, know the limit dimensions, and apply standard techniques to stay fast and within quota.

What makes a request slow

Total latency is roughly time to first token (TTFT) plus generation time. TTFT grows with input size, because the model must read the whole prompt before it can respond, and generation time grows with output length, because tokens are produced sequentially. Model tier matters as well: Haiku generates fastest, Opus slowest. Extended thinking adds reasoning tokens before the visible answer.

Practical levers, in order of impact:

1. Stream. It does not shorten the request, but users see output immediately and TTFT becomes the perceived latency.
2. Cache the prefix. Cached tokens are not re-processed, so a large cached system prompt cuts TTFT substantially.
3. Shorten input and cap output. Fewer tokens in, fewer tokens out.
4. Pick a faster tier for routes that do not need the most capable model.
5. Reduce thinking effort where supported, or turn thinking off for simple tasks.
6. Parallelise independent calls instead of chaining them.

Rate limit dimensions

Limits are applied per organisation (and optionally per workspace) and per model, along three axes:

Limit	Meaning
RPM	Requests per minute
ITPM	Input tokens per minute
OTPM	Output tokens per minute

Organisations move through usage tiers as cumulative spend grows, with higher tiers unlocking higher limits; the current tiers and numbers are on the rate limits page at docs.anthropic.com. Token-based limits are estimated at request time; for OTPM the API estimates from max_tokens and reconciles after the response, so a very large max_tokens on every request can throttle you even if actual output is short. On most models cache reads do not count towards ITPM. The Batches API and the count_tokens endpoint have their own separate limits.

Every response carries headers you can read to stay ahead of throttling:

retry-after: 7
x-ratelimit-limit-requests: 4000
x-ratelimit-remaining-requests: 3996
x-ratelimit-reset-requests: 2026-10-19T10:15:00Z
x-ratelimit-limit-tokens / -remaining-tokens / -reset-tokens

Error codes: retry or fix?

Status	Type	Action
400	invalid_request_error	Fix the request; retrying will not help
401	authentication_error	Fix the key
403	permission_error	Fix permissions or access
404	not_found_error	Fix the model ID or path
413	request_too_large	Reduce input size
429	rate_limit_error	Wait retry-after seconds, then retry with backoff
500	api_error	Retry with backoff
529	overloaded_error	Retry with backoff; consider another model or time

The 4xx group (except 429) are deterministic failures: the same request will fail again. The 429, 500 and 529 group are transient and should be retried with exponential backoff and jitter.

Retries in practice

The SDKs retry 408, 409, 429 and 5xx responses and connection errors automatically, with exponential backoff, two times by default. Raise max_retries for background jobs and lower it for latency-sensitive paths where you would rather fail fast and fall back.

client = anthropic.Anthropic(max_retries=5)
try:
    resp = client.messages.create(...)
except anthropic.RateLimitError as e:
    wait = int(e.response.headers.get("retry-after", "30"))
    schedule_retry(after=wait)
except anthropic.InternalServerError:
    use_fallback()

const client = new Anthropic({ maxRetries: 5 });
try {
  await client.messages.create({ ... });
} catch (err) {
  if (err instanceof Anthropic.RateLimitError) {
    const wait = Number(err.headers?.["retry-after"] ?? 30);
  }
}

Honour retry-after rather than retrying immediately; hammering a throttled endpoint prolongs the throttle.

Throughput architecture

• Client-side concurrency control. Use a semaphore or queue sized to your RPM and TPM so you shape traffic instead of discovering the limit through 429s.
• Batches for bulk. Move non-interactive volume to the Message Batches API, which has separate limits and a discount.
• Spread across workspaces when teams compete for the same organisation quota and you want isolation.
• Request a limit increase through the Console when you have real demand; tiers are not the only path.
• Timeouts. Set a timeout that reflects the expected generation time, remembering total wall-clock can reach timeout × (max_retries + 1).
• Prefer streaming for long outputs; it avoids both HTTP timeouts and poor perceived latency.

Watching for overload

A 529 means Anthropic's capacity is constrained, not that you did something wrong. Back off, retry, and if it persists route to a different model tier or defer the work. Check the public status page before escalating.

Exam focus

• Latency is input processing (TTFT) plus sequential output generation; streaming, caching, shorter prompts and smaller models reduce it.
• Rate limits are RPM, ITPM and OTPM per organisation and model, rising with usage tier; OTPM is estimated from max_tokens.
• 429 is retryable after retry-after; 500 and 529 are retryable with backoff; other 4xx errors need a fix, not a retry.
• SDKs retry transient errors automatically (max_retries default 2) with exponential backoff.
• x-ratelimit-* headers report limits, remaining quota and reset times on every response.
• Batches and count_tokens have separate rate limits, and cache reads generally do not count towards input token limits.