AI coding tools make it easier to produce code. That is useful, but it is not the same as making software engineering easier end to end.
When implementation gets faster, the bottleneck can move. Teams may spend less time typing and more time deciding what should be built, checking whether generated changes are correct, understanding unfamiliar code, integrating components and diagnosing failures. If leaders measure only code produced or accepted suggestions, they can miss the actual cost of delivery.
The important question is not whether AI can write code. It is whether the whole engineering system can deliver valuable, maintainable software more reliably.
The engineering job is moving up the stack
AI can be particularly useful when a task is well specified, the relevant context is available, and the result can be checked cheaply. Boilerplate, test scaffolding, documentation drafts, mechanical refactoring and exploration of unfamiliar code are common examples.
But the tools do not remove the need to define the problem, choose the right boundaries, understand the system’s constraints or decide whether a proposed change is safe. In ambiguous work, a fast implementation of the wrong idea is still waste.
As code generation becomes cheaper, four capabilities become more valuable:
- Problem framing: turning a vague request into a testable outcome and explicit constraints.
- System judgement: understanding architecture, dependencies, failure modes and long-term maintenance costs.
- Verification: designing tests and evaluations that can distinguish a plausible answer from a correct one.
- Integration and ownership: making sure the change behaves properly in the real product and that someone is accountable for it.
Measure the work around the code
Lines of code, prompts sent and AI acceptance rates are activity measures. They may help explain how a tool is being used, but they are weak proxies for customer value.
For a four-to-six-week experiment, establish a baseline and track a small scorecard:
- Lead time: how long does a well-defined change take from ready to production?
- Review rework: how often does a change need substantial revision after review?
- Escaped defects: do more defects reach users after AI-assisted changes?
- Change failure rate: are releases causing more incidents, rollbacks or hotfixes?
- Developer effort: how much time goes into prompting, checking, debugging and rewriting—not just initial generation?
- Outcome: did the change solve the user or business problem it was meant to solve?
Compare similar tasks and teams where possible. Record task complexity and experience level. Avoid declaring success because one easy task was completed quickly or because a team used the tool more often.
Design the workflow, not just the prompt
Give the model a bounded task, the relevant repository context, clear acceptance criteria and access only to the tools it needs. Require tests and make review expectations explicit. For higher-risk changes, use smaller pull requests and stronger automated checks.
Also measure the verification burden. If a developer spends ten minutes generating a change and forty minutes discovering subtle errors, the workflow may not be an improvement. Conversely, a tool that reduces repetitive work without degrading quality can be valuable even if it does not dramatically change headline throughput.
A practical 30-day experiment
- Week 1: choose one workflow, define comparable task types and capture the baseline.
- Weeks 2–3: use AI consistently on that workflow. Keep existing quality gates and record generation, review, rework and debugging effort.
- Week 4: compare results, inspect failures and ask developers what changed in their work.
Set a hypothesis before starting—for example, “AI will reduce median cycle time for routine test additions by 15%, without increasing review rework or escaped defects.” Treat that number as a target to test, not a promised result.
The leadership shift
The best engineering organisations will not be the ones that generate the most code. They will be the ones that learn where AI genuinely removes friction, build reliable verification around it, and redirect human attention towards decisions that require context and judgement.
AI can change who—or what—does the first draft. It does not change the responsibility to ship software that works.
A question for engineering leaders: if AI wrote twice as much code for your team next quarter, which metric would tell you whether that was actually good news?