How to Measure Developer Productivity With AI

I disagree with a tempting idea about AI coding tools: if developers can produce more code, the team must be more productive.

Code volume was never a good proxy for software value. It becomes even less useful when a coding agent can generate a large patch in minutes. The organization may see more pull requests, commits, or completed tickets while reviewers absorb more work, defects travel faster, and the number of systems requiring maintenance quietly grows.

The opposite conclusion is also too simple. AI-generated code is not automatically low quality, and using an assistant is not evidence of weak engineering. These tools can remove repetitive work, help people explore unfamiliar code, draft tests, explain APIs, and shorten some implementation tasks. The effect depends on the task, the developer, the codebase, and the operating system around the work.

That leaves engineering leaders with a harder question: how should developer productivity be measured when AI changes both the speed of creation and the cost of verification?

My answer is a balanced scorecard. Measure whether the team delivers useful outcomes, protects quality, improves flow, sustains healthy collaboration, and learns where AI helps. Never collapse those signals into a leaderboard for individuals.

Start With the Scorecard, Not the Tool Dashboard

An AI assistant’s dashboard can report suggestions, acceptance rates, active users, and generated lines. Those figures describe tool activity. They do not establish that customers received value or that engineering became more effective.

Use a five-part scorecard instead:

No row is sufficient alone. A faster flow that damages quality is not durable. Excellent quality on work nobody needs is not productivity. High activity supported by exhaustion is borrowed capacity. Learning without delivery is research, which can be valuable, but should be named honestly.

This structure is consistent with the SPACE framework, developed by researchers from GitHub, Microsoft Research, and the University of Victoria. SPACE treats developer productivity as multidimensional: satisfaction and well-being, performance, activity, communication and collaboration, and efficiency and flow. Its central warning remains especially relevant in the AI era: productivity cannot be represented by one metric.

Generated Code Is an Input, Not an Outcome

Lines of code, commits, pull requests, and ticket counts are attractive because they are available. They are also easy to misunderstand.

A developer who removes 2,000 lines may simplify a service and lower its operating cost. Another may add 2,000 lines produced by an agent and create a new maintenance obligation. A senior engineer may spend a day preventing the wrong feature from being built. A platform team may improve a shared deployment path, allowing dozens of teams to work faster without generating a visible product feature of its own.

Activity measures can still help diagnose a workflow. A sudden increase in pull-request size might explain slower review. A fall in commits might reveal blocked work. The problem begins when activity becomes a performance target. People then have a reason to split commits, inflate tickets, avoid deletion, or choose visible tasks over valuable ones.

AI makes gaming easier even when nobody intends to game. If management celebrates code generated or tickets closed, tools will help people maximize exactly those numbers. The metric changes behavior before leaders understand whether that behavior is useful.

Do not ask, “How much more code did AI help us produce?” Ask, “Which constraint moved, and what happened elsewhere in the system?”

Measure the Entire Change, Including Verification

Coding is one part of software delivery. A change still needs context, design, review, tests, security checks, deployment, observation, documentation, and long-term ownership. AI may reduce effort in one stage while increasing it in another.

Google Cloud’s 2025 DORA research describes AI-assisted development as an amplifier of an organization’s existing strengths and weaknesses. That is a more useful model than treating a license as a guaranteed productivity gain. Clear requirements, fast feedback, good internal platforms, small batches, and reliable tests give AI output a stronger path to production. Confusing architecture and weak controls can become faster sources of rework.

DORA’s March 2026 discussion of AI tensions in the software lifecycle adds an important detail: time saved during initial creation is often reallocated to auditing and verification. It also reports a tension between higher throughput and delivery instability. A local gain can therefore coexist with a system-level cost.

For each class of work, measure the path from a meaningful request to a safely operating change:

1. Record the baseline lead time, including waiting and review.
2. Identify the stage where AI is expected to help.
3. Track whether work moves to another queue rather than disappearing.
4. Compare quality and operational signals before and after adoption.
5. Ask developers and reviewers what changed in cognitive load.

Suppose an agent cuts implementation from eight hours to four, but review increases from one hour to three and follow-up corrections add two. The coding stage improved; total effort did not. That is not a failed experiment. It is useful evidence about where the workflow needs better task selection, tests, context, or review rules.

Productivity Effects Depend on the Work

Broad claims about AI coding usually hide the task distribution.

Generating a small adapter for a well-documented API differs from changing a mature authorization system. Drafting unit tests for a pure function differs from diagnosing an intermittent production failure. A developer new to a framework may benefit from explanations that distract an expert who already understands the repository. An agent can perform well in a codebase with fast tests and clear boundaries yet struggle where expected behavior lives in people’s memories.

Research results should be read with those boundaries intact. METR’s 2025 randomized study of experienced open-source developers found that participants working on their own repositories took longer with the AI tools tested at that time. The result was striking, but the researchers explicitly warned against generalizing it to most developers or most software work. In a February 2026 update, METR said newer tools likely provided more acceleration, while selection effects and parallel agent use made the size of that effect difficult to estimate reliably.

That changing evidence is a reason to run local experiments, not a reason to wait for one universal verdict.

Segment the work into categories such as routine implementation, test creation, code understanding, migration, refactoring, incident diagnosis, and high-risk changes. Compare similar tasks. Record relevant context: repository maturity, developer familiarity, test speed, review standard, and tool configuration. A team can then say, “AI helps us draft routine integrations but adds little to incident diagnosis,” which is far more actionable than “AI improved productivity by 20 percent.”

Do Not Turn the Scorecard Into an Individual Ranking

Most valuable software is produced by a system of people. Product decisions, design discussions, shared libraries, reviews, documentation, incident response, mentoring, and infrastructure all influence what an individual developer can deliver.

Individual rankings break that system into misleading fragments. They penalize people who take difficult assignments, help colleagues, review risky changes, reduce code, or prevent incidents. They can also discourage engineers from reporting problems with AI output. If accepting more suggestions improves a personal score, careful rejection starts to look like poor performance.

Use the scorecard primarily at team and value-stream level. Discuss individual performance through evidence and context: the difficulty and importance of the work, technical judgment, collaboration, ownership, learning, and impact. Numbers may inform that conversation, but they should not calculate the answer.

This is where Measure AI Work Without Losing Leadership Judgment provides the broader principle: a dashboard can describe system behavior, but it cannot decide what the work should be or evaluate every form of contribution.

Privacy also matters. Detailed telemetry about editor activity, prompts, keystrokes, or time can damage trust and create security risks. Collect the minimum data needed to improve the workflow. Explain what is collected, why it is used, who can see it, and when it is deleted. Do not quietly repurpose improvement telemetry for performance management.

Run a Six-Week Measurement Experiment

A measurement program does not need to begin with an enterprise dashboard. Start with one team, one workflow, and one decision the evidence will support.

Week 1: Define the outcome. Choose a real problem such as reducing lead time for small customer fixes without increasing escaped defects. Name what is outside the experiment. Do not use “increase AI adoption” as the outcome.

Week 2: Establish a baseline. Review recent comparable work. Record lead time, review wait, change-failure or rollback patterns, developer experience, and any user outcome available. Imperfect baseline data is better than an invented precision target.

Week 3: Choose task boundaries. Decide where AI may be used and where extra review is required. For sensitive code, permissions, customer data, or production actions, set explicit constraints. Keep a way to complete the work without the tool.

Weeks 4 and 5: Observe the whole flow. Compare task categories rather than pooling everything. Sample pull requests for maintainability and security. Ask authors and reviewers where time moved. Record failure patterns, not private prompt histories.

Week 6: Make a decision. Expand the uses that show balanced gains. Change guidance where review or rework increased. Stop using the tool for tasks where it adds friction or risk. Document uncertainty and schedule another check as tools and team practices change.

The decision may be narrower than leaders expected. That is a strength. “Use the agent for test scaffolding and routine migrations, with human ownership of architecture and final review” is a workable policy. “Everyone should use AI more” is not.

If faster generation starts consuming review or maintenance capacity, Protect Reliability While Shipping AI Faster offers a complementary reliability-floor approach. It helps leaders decide when extra delivery demand is no longer safe to absorb.

Watch for Four Measurement Failure Modes

The numerator has no denominator. A team reports 300 AI-assisted pull requests but not how many were reverted, substantially rewritten, or never used. Report completion together with quality and total effort.

The baseline changes during the test. The AI-enabled period includes easier work, a quieter release cycle, or more experienced developers. Record these differences instead of presenting a clean causal claim.

People optimize the observation. Once acceptance rate becomes a target, acceptance rises. Treat tool metrics as diagnostic signals and keep them away from rewards.

The average hides the risk. Ten quick low-risk changes can conceal one harmful permission error. Segment by consequence and review severe failures separately from average performance.

These are feedback-loop problems as much as measurement problems. Fix the Feedback Loops Behind AI Team Failure explains how dashboards, incentives, and review rituals can reward behavior that contradicts the stated goal.

Teaching Reinforces the Same Lesson

When I teach data and AI, learners naturally focus first on producing the visible result: code that runs, a dashboard that renders, or a model that returns an answer. The more important development comes when they can test the result, explain a tradeoff, investigate a failure, and make the work reproducible.

The same distinction applies inside engineering organizations. Output is easy to display. Capability is revealed by what happens around that output: whether the team chose the right problem, checked the work, learned from failure, and can operate the result responsibly.

AI does not make measurement unnecessary. It makes thoughtful measurement more important because visible production has become cheaper. Leaders now need stronger ways to distinguish activity from outcomes and local speed from system performance.

Use code volume and tool adoption only as clues. Measure the complete flow of useful work. Balance outcome, quality, flow, team health, and learning. Keep the evidence at the level where software is actually delivered: the team and its operating system.

The goal is not to prove that AI is productive. The goal is to learn where it helps your team produce better software—and where human judgment, stronger engineering systems, or a different tool remains the better answer.