Measurement
Measuring engineering work when AI writes code
When a tool can produce a thousand lines in a minute, counting output tells you almost nothing. Engineering leads need measures of delivery, quality and team health that AI can't inflate.
Metrics that mislead
- Lines of code or commits. Always weak, now meaningless. More code is often more to maintain.
- AI suggestion acceptance rate. It measures how often people press accept, not whether the result was right. Useful to a tool vendor, not as a team goal.
- Pull requests per engineer. Easy to raise by splitting work or generating trivial changes.
- Story points completed. An estimation aid, not a productivity measure, and easily re-scaled.
Any metric that becomes a target will be optimized. With AI tools, optimizing output metrics has never been easier, which is exactly why they shouldn't be targets.
Delivery: the DORA measures
The DORA research program's delivery measures remain a sound starting point because they reward shipping safely, not shipping a lot:
| Measure | What it tells you |
|---|---|
| Deployment frequency | How often the team delivers change to users |
| Lead time for changes | How long from commit to running in production |
| Change failure rate | How often a deployment causes a failure that needs remediation |
| Time to restore service | How quickly the team recovers when a change fails |
If AI tools raise deployment frequency while change failure rate climbs, the team is going faster, not better. Watch the pair together.
People and flow: the SPACE framework
The SPACE framework argues that developer productivity can't be captured in one number. It looks across Satisfaction and well-being, Performance, Activity, Communication and collaboration, and Efficiency and flow. Use a few measures from different dimensions, and include what engineers themselves report.
Measures worth adding in the AI era
- Review load and review time. If change volume rises faster than review capacity, quality will follow it down.
- Rework and reverts. Changes reverted or substantially rewritten within weeks of merging.
- Escaped defects and incidents, linked back to the changes that caused them.
- Onboarding time to first meaningful change, and whether new engineers can explain it.
- Engineer-reported friction: short, regular surveys on what slows the team down.
Running an honest AI tool evaluation
- Decide what outcome you expect before rollout, and measure a baseline first.
- Compare like with like: similar work, similar teams, over enough time to see quality effects, not just speed.
- Count the costs: licenses, review time, incidents and security work, not just time saved drafting.
- Ask engineers. Their experience of where the tool helps and where it hurts is data.
Use metrics on systems, not individuals. Team-level delivery measures improve conversations about process. Individual output rankings damage trust and are especially easy to game with AI tools.
Last reviewed 2026-09-17