What To Track When Measuring AI Coding’s Return

🔍 Read the full analysis: What To Track When Measuring AI Coding’s Return on ThorstenMeyerAI.com

Age 18–24?Offer from Amazon

Prime made for students and young adults

• Fast, free delivery for dorm and study essentials
• Prime Video and Amazon Music included
• Member-only deals

Try Prime for Young Adults →Free trial for eligible 18–24 year olds

As an affiliate, we earn on qualifying purchases.Hide

As companies set 2027 engineering budgets, a survey of 27 workplaces finds AI coding spending ranges from $150 to about $3,000 per developer per month, with no shared measure of what those costs return. The findings, reported by Gergely Orosz in The Pragmatic Engineer on Oct. 20, show that pull-request output varies widely and that companies need to measure accepted work, rework and delivery outcomes—not just tool consumption.

The survey includes accounts from engineering leaders and senior engineers, plus aggregate data from AI code-review vendors CodeRabbit and Gitar. Its spending examples span a publicly traded company of 1,500 engineers with a $150 monthly cap and a two-person European .NET agency, Ix Technologies, spending about $3,000 per developer. Other companies set caps between $1,000 and $4,000, with typical reported usage around $1,000. These are observations from the workplaces surveyed, not a representative estimate of all companies.

Reported output measures also differ. CodeRabbit says developers on its platform created 113% more pull requests per week than a year earlier. Microsoft’s study of tens of thousands of engineers during an early-2026 rollout found adopters merged about 24% more pull requests over four months than they otherwise would have. The studies measure different groups and periods; Microsoft’s researchers cautioned that a merged pull request is not the same as delivered value.

In the surveyed workplaces, some teams reported flat pull-request counts while others reported increases as high as five to ten times. One company said frequency stayed flat while pull requests became three times larger. The source argues that AI appears most useful for tasks such as customer-issue triage, understanding older code, prototyping and backlog work on simple, consistent technology stacks. Those examples are company accounts, not controlled comparisons across teams.

Budgets Need Outcome Measures

The spending spread and mixed output figures make it difficult for leaders to use licenses, tokens or pull-request totals as a stand-in for return. A team can generate more code without shipping more useful work, and larger changes may carry additional review and maintenance costs. The source recommends tracking cost per accepted change, rework, change-failure rate and whether completed stories outpace incoming work.

Quality is part of that calculation. The source cites Veracode’s latest test, in which 44% of AI coding tasks produced a known vulnerability. That result signals a need to track security and review alongside speed; it does not establish the vulnerability rate for every company or AI coding tool. If review capacity does not keep pace with code production, higher activity may not translate into safer or faster delivery.

Why AI Coding Costs Are Shifting

Orosz’s Oct. 20 report arrives as companies prepare 2027 budgets and some discounts on AI subscriptions are due to end, according to companies quoted in the report. Several said the loss of subsidies could raise bills five- to tenfold unless they optimize usage. The report also says spending by companies on API plans peaked in June and July and had roughly halved by the time of publication, which Orosz attributed to a move to open models and the release of Opus 5.5.

Billing changes add pressure to measure consumption. GitHub changed every Copilot plan to usage-based billing on June 1, charging for tokens at each model’s API rate and removing the fallback to a cheaper model. GitHub said agentic use brings higher computing and inference demands. The source also describes spending caps as common but often easy to get around, making actual usage and business outcomes harder to connect.

Controlled research cited in the source is mixed. METR’s early-2025 study found experienced open-source developers took 19% longer with AI. A February follow-up estimated an 18% speedup for returning developers and 4% for new developers, but both confidence intervals included zero; METR also called the data unreliable because some participants held back tasks they did not want to do without AI. Those results do not settle how AI performs across commercial engineering teams.

“a question that few companies seem able to answer – at least for now.”

— Gergely Orosz, The Pragmatic Engineer

The Return Remains Hard to Isolate

The findings do not establish a single, general return on investment for AI coding. The 27 workplaces were not presented as a representative sample, while vendor metrics describe activity on their own platforms. Company accounts, controlled studies and self-reported productivity measure different things and cannot be directly combined into one productivity estimate.

It is also unclear how much extra code becomes useful, maintained software, or whether reported gains persist after teams adapt their workflows. The source’s account of spending changes does not establish that model changes alone caused lower bills. Companies will need to compare costs and outcomes over time, while accounting for task complexity, review effort, security findings and work that would have been completed without AI.

Track Results Through Budget Reviews

As 2027 budgets are finalized, engineering leaders can establish a baseline for spending and delivery before expanding or restricting AI tools. The source’s proposed measures—cost per accepted change, rework, change failures and stories closed versus incoming—can be compared across teams and time periods, with task types recorded so that simple backlog work is not treated as equivalent to complex legacy changes.

Companies should also watch what happens when subscription discounts expire and usage-based billing changes alter incentives. The next useful evidence will show whether lower or higher spend changes accepted work, delivery speed, defects and security review—not merely how many tokens or pull requests teams produce. The source does not provide a common measurement standard or a confirmed timeline for such follow-up data.

Key Questions

How much are companies spending on AI coding tools?

The 27 workplaces in The Pragmatic Engineer’s report described budgets from $150 to about $3,000 per developer per month. Those figures are survey examples, not an industry-wide average.

Do more pull requests prove AI coding is paying off?

No. More pull requests indicate increased activity, but they do not show whether changes are accepted, useful, secure or costly to maintain. Microsoft’s study authors specifically cautioned that a merged pull request is not the same as delivered value.

Which AI coding measures does the source recommend?

It recommends tracking cost per accepted change, rework, change-failure rate and whether teams close more stories than they receive. Those measures connect spending with delivery and quality more directly than token use alone.

Where did surveyed teams report the clearest benefits?

Accounts in the report describe gains in triage, code archaeology, prototyping and backlog cleanup on simple, uniform technology stacks. These are reported examples, not proof that every team will see the same results.

What is still unknown about AI coding’s return?

The report does not establish a universal return or show how much increased activity becomes durable business value across companies. The evidence varies by task, team and measurement method, and the survey is not presented as representative of the whole industry.

Source: ThorstenMeyerAI.com