Lesson 7.3
Measuring Adoption and Impact
Which metrics show whether a Claude Code rollout is being used and is paying off, where the data comes from, and how to avoid the vanity measures and attribution traps that discredit productivity programmes.
Leadership funds a Claude Code rollout on a promise of productivity. A year later it will ask whether the promise was kept. The professional architect designs measurement into the programme from the start, chooses metrics that reflect real outcomes, uses the telemetry the platform provides, and reports honestly about what can and cannot be attributed.
Three layers of metrics
| Layer | Question | Example metrics |
|---|---|---|
| Adoption | Are people using it? | Weekly active users, sessions per user, retention after 30 and 90 days, share of teams onboarded |
| Usage | How are they using it? | Tokens and cost per developer, tool acceptance rate, share of sessions using subagents or skills, permission denials, model mix |
| Outcome | Is it making a difference? | Cycle time from first commit to merge, review turnaround, pull request throughput, defect and incident rates, developer satisfaction, onboarding time for new engineers |
Adoption and usage are leading indicators and are easy to collect. Outcomes are what leadership actually wants and are harder to attribute. Report all three, and be explicit about which is which.
Data sources
- Telemetry export. Claude Code can emit metrics and events through OpenTelemetry to your observability stack: session counts, active time, tokens and cost by model, lines of code changed, commits and pull requests created, tool decisions and API errors. Enable it through managed settings so coverage is complete and attributes (team, department) are consistent.
- Console and plan dashboards. The Console provides analytics for Claude Code usage on API-based organisations, and Team and Enterprise plans provide usage views; the Admin API exposes usage and cost reports for programmatic analysis.
- Engineering systems. Git, code review and CI data provide cycle time, throughput, review metrics and defect rates. Join them to telemetry by team, not by individual.
- Surveys. Short, regular developer surveys capture satisfaction, perceived time saved and friction. Perception is a real outcome for retention and recruitment.
Establish the baseline and a comparison
Impact is a difference, so measure the same outcome metrics before the rollout and, where possible, in teams that have not yet adopted. A staggered rollout gives you a natural comparison group; use it. Where no comparison exists, report trends with the caveat stated. Avoid before-and-after claims on a single team during a period when other things also changed.
Metrics to be careful with
- Lines of code. Easy to collect, easy to inflate, weakly related to value. Report it as usage, never as outcome.
- Suggestion acceptance rate. Useful for tuning, misleading as productivity; a high rate may mean uncritical acceptance.
- Tokens or cost per developer. A cost metric, not a productivity metric; pair it with outcomes to get cost per unit of value.
- Self-reported time saved. Optimistic; triangulate with cycle time.
Any metric that becomes a target will be gamed. Report a balanced set and change the emphasis periodically.
Quality guardrails
Faster is not better if quality falls. Track review rejection rates, post-merge defects, incident counts and security findings on code changed with assistance versus without. A rollout that lifts throughput while raising defects has shifted work to reviewers and operations; the net may be negative. Make quality guardrails part of the programme's definition of success.
Cost governance
Attribute spend to teams through workspaces or telemetry attributes, set budgets and alerts, and report cost alongside outcomes as cost per merged change or per engineer-month. Investigate outliers: very high spend per developer often indicates a workflow problem (long sessions without compaction, large files repeatedly read) that a skill or CLAUDE.md change can fix. Share the fixes through the champions network.
Reporting cadence
Monthly to the programme sponsor: adoption, usage, cost and friction. Quarterly to leadership: outcomes against the baseline and the business case, quality guardrails, and decisions requested (expand, invest in skills, adjust policy). Keep the dashboards live so that the quarterly report is a narrative over numbers everyone can already see.
Linking back to the business case
The business case promised specific benefits. Report actuals against those, not against new metrics chosen after the fact. Where a benefit has not materialised, say so and explain what the data suggests (adoption gap, workflow fit, tooling limits) and what will be tried. Honest reporting is what earns the programme its next phase.
Exam focus
- Measure three layers: adoption (leading), usage (diagnostic), outcome (what leadership wants), and label them as such.
- Data comes from OpenTelemetry export enabled through managed settings, Console and plan dashboards, the Admin API, engineering systems and surveys.
- Impact needs a baseline and a comparison group; staggered rollouts provide one.
- Lines of code, acceptance rate and tokens per developer are usage or cost metrics, not productivity outcomes.
- Quality guardrails (defects, review rejections, incidents) are part of the success definition.
- Report actuals against the original business case on a fixed cadence, including misses.
Check yourself
5 questions
1.An engineering director asks for objective data on how Claude Code is being used across teams: sessions, active users, token consumption, cost and code accepted. Which approach provides this systematically?
2.A rollout report celebrates a 300% increase in lines of code produced per engineer since Claude Code was introduced. The CTO asks the architect whether this proves the investment is paying off. Which response is MOST rigorous?
3.Finance wants to understand the return on Claude Code spend. The architect has telemetry on token usage and cost per team, and delivery metrics from the engineering platform. Which analysis is MOST useful for the finance audience?
4.Telemetry shows that 85% of engineers have used Claude Code at least once, but the rollout lead wants to understand whether it is genuinely embedded in daily work and where it falls short. Which measurement plan is MOST appropriate?
5.Six months after rollout, telemetry shows Claude Code spend concentrated in one team whose usage is ten times the average, while their delivery metrics are unchanged and their incident rate has risen. Which interpretation and action is MOST appropriate?