AI-Assisted Development: How to Supplement Output Metrics with Review Friction

by
Table of contents

🌏 閱讀中文版本

If a team sets its AI-assisted development goal as “maximizing code output,” whether the system bottleneck shifts to review, increasing maintenance cost and delivery friction, depends on changes in the arrival rate of changes and the workload of each review. When evaluating return on investment (ROI), the most intuitive choice is often a proxy metric such as generated lines of code (LOC). That reflects an engineering-management reality: the metrics that are easiest to quantify are often adopted first.

One hypothesis worth validating is that generative AI models, when operating without global context, may tend to produce solutions that are verbose, overly defensive, or heavily commented. If that assumption holds in a specific workflow, it may increase apparent output volume while raising the cost of validation during review in the actual engineering pipeline.

When discussing output, it is important to distinguish generated LOC, submitted changed LOC, net new LOC, and churn. In practice, accurate measurement requires using the PR or task as the unit of analysis. One evaluation design worth validating is to define how AI exposure is labeled, for example through IDE telemetry or PR labels, and explicitly treat it as a treatment or explanatory variable. Each outcome metric should use its own appropriate denominator; LOC may be used only for stratification or sensitivity analysis and must not replace AI/non-AI grouping. The evaluation design also needs to distinguish three levels: directly observing metric trends is descriptive monitoring; matching with covariates such as task difficulty or risk strata is confounding adjustment, which can estimate conditional associations but is still insufficient to identify causal effects; and rigorously estimating the ROI of AI adoption through causal inference requires designs such as randomized controlled trials (RCTs) or difference-in-differences (DiD). If these detailed data and experimental conditions are not yet available, the following reasoning serves as an initial hypothesis for building a measurement framework.

According to GitClear’s analysis of roughly 150 million lines of code changes from 2020 to 2023, code churn, defined in the report as lines overwritten, reverted, or deleted within two weeks of submission divided by total changed lines in that submission, showed an upward trend. This is only a concurrent signal, not causal evidence that AI use increased churn. But it offers a useful perspective: output volume is not the same as delivery value. If code-generation throughput exceeds a team’s review and integration throughput, the system’s marginal benefit may decline quickly.

In software engineering, added code expands the maintenance surface. Whether that investment is worthwhile depends on delivery value and lifecycle cost, while the cognitive load of reading and validating code is often underestimated. Human engineers working together omit obvious steps based on shared context. If AI tends to enumerate every detail without decision context, that indiscriminate completeness can become substantial friction during review.

Key Insight: Friction Cost and Net Benefit

Key Insight Whether the productivity bottleneck in AI-assisted development shifts toward delivery friction depends on system conditions. In practice, capacity, overload, and bottlenecks must use consistent definitions: as utilization approaches 1, waiting time may increase nonlinearly; utilization greater than or equal to 1 means the queue cannot be processed stably; and a bottleneck is the stage that constrains end-to-end throughput, not simply a stage under overload. A higher arrival rate or longer review time can both create overload. Review may become the bottleneck if either of the following mechanisms occurs, though other causes may also exist:

  1. A higher change arrival rate: when submitted-change throughput exceeds the team’s review and integration capacity.
  2. Longer time per review: when AI-generated code increases the complexity of reading, understanding, and validating changes. These two states need independent criteria: when review constrains end-to-end throughput, it is a system bottleneck; when incremental costs, such as added friction and risk, exceed incremental delivery value, the system’s net benefit becomes negative. Either can hold on its own and they should not be conflated.

To explain this mechanism, we can build a conceptual model:

Net Value = Delivery Value − (Generation Cost+Review Cost+Rework Cost+Operation Cost+Expected Risk Loss)

This is a framework for clarifying trade-offs. Before measuring in practice, delivery value, generation, review, rework, post-release operational cost, and expected risk loss need to be converted into comparable units. To avoid double counting, each cost item needs a mutually exclusive boundary, and, where possible, should first be converted into a common unit, such as converting engineering hours into financial value using a cost rate. Otherwise, they should be evaluated separately. When some metrics cannot reasonably be converted into one common unit, use a balanced scorecard with quality guardrails instead.

Operation Cost refers to incurred day-to-day maintenance cost: escaped defects, fixes, and incident costs that have already occurred within the observation window all belong here. Expected Risk Loss should be used only for events that have not yet occurred but are predicted from defect-rate estimates and weighted by probability. Rework Cost refers specifically to the cost of changes sent back for rewriting in CI/CD and review before release.

There is an important logical boundary here: an increase in review time necessarily raises the cost side, but does not necessarily make overall net benefit negative. Review friction can rise while net benefit remains positive, provided that the incremental delivery value brought by AI, in common units, or the development time it saves is sufficient to cover the added review, rework, and potential risk costs.

This leads to an operational decision rule: to assess whether AI-assisted development delivers a material improvement, all deltas must be calculated against the same baseline, and the signs of every cost item in the full formula must remain consistent. Specifically, Time Saved counts only as a reduction in Generation Cost, a negative delta; Cycle Time is an independent delivery-outcome metric. When calculating total incremental net benefit, units must be consistent, converting engineering hours to financial units with a cost rate. Only when the increase in delivery value, ΔDelivery Value, remains greater than the total incremental cost of generation, review, rework, operations, ΔOperation Cost, and expected risk can we conclude that net benefit has improved at the system level. If units cannot be normalized, they should not be forcibly added together. Evaluate benefits and costs separately instead.

This forms a testable scenario hypothesis: when a small change to one module produces a disproportionate amount of supporting code because of the generation strategy, while also creating frequent integration rework, its review and rework costs may offset the value of the change itself. Validating that hypothesis requires measuring review cost, rework cost, and delivery value with the same baseline and common units before drawing a conclusion.

Using delivery friction as the higher-level concept, we can break it into three dimensions for evaluating system balance:

  1. Review effort: the human time and cognitive load required to process one change.
  2. Integration friction: the resistance and rework cost a change encounters before entering the main branch.
  3. Quality risk: the maintenance cost a change may create after release.

Redefining Productivity: A Contextual Coexistence Framework

Addressing this engineering question requires shifting measurement from output volume to system outcomes. DevEx research points to cognitive load and feedback loops as underlying mechanisms of productivity. Based on those mechanisms, this article uses review, defects, and CI as operational dimensions: CI and review waiting time correspond to feedback loops, active review effort corresponds to cognitive load, and defects act as a quality guardrail. Note that these specific metrics are selected for this article; they are not directly validated by the DevEx source.

It must be stated clearly that these three metrics monitor only cost-side friction. They cannot, on their own, answer questions about productivity or ROI. To evaluate full net benefit, the benefit side needs benefit measurement as well: label Accepted Delivery and End-to-End Cycle Time explicitly as delivery outcome metrics, rather than calling them Delivery Value. For delivery value on the benefit side, use accepted scope, business outcomes, or value-weighted delivery volume as proxies.

If those proxies can be reliably converted, first normalize them and all costs into financial value, with engineering hours multiplied by a cost rate, then calculate total incremental net benefit. If reliable conversion is not possible, present them separately. Use delivery-outcome metrics as independent reference points, alongside a cost balanced scorecard and quality guardrails.

This shift requires teams to establish new criteria. Early in adoption, the first task is to establish a baseline, not to pursue a perfect global metric. Start by observing the stages with the most visible friction. Here is a practical three-dimensional measurement framework:

1. Code Review Effort

This is a reference metric for the comprehensibility of AI-generated code. If the writing time saved by AI is fully offset by added reading and understanding effort, its net benefit requires closer examination.

  • How to measure: Separate Review Lead Time from Active Review Effort. PR timestamps capture waiting time only; in practice, accurately evaluating active effort usually depends on review-tool activity records or sampled time tracking. Normalize by change size to control variables.
  • How to observe: If average review time rises materially after introducing AI, that is a signal worth breaking down further. Before attributing the friction to the generation strategy, first account for confounding variables such as higher task difficulty, changed risk levels, and reviewer load.

2. Escaped Defect Metrics

This is a reference metric for the correctness and boundary handling of AI-generated code. Review interception rate can also serve as a supporting measure, showing whether AI-generated code is being effectively validated during review.

  • How to measure: Use a consistent observation window, for example the first 14 days after production deployment. Teams can establish metrics that fit their context, but defects cannot be measured by count alone; they need weighting by actual remediation cost and severity. Simply counting escaped defects can be misleading. The primary metric can be expected defect loss per accepted delivery unit, task, PR, or value-weighted scope. Changed LOC should be used only for size stratification or sensitivity analysis, avoiding the dilution of defect density through higher code output. Clear task labeling is needed to establish comparable baselines.
  • How to observe: Before comparing AI-generated code with an existing baseline, predefine a non-inferiority margin, required sample size, and severity guardrails. The absence of a statistically significant difference cannot substitute for an acceptable equivalence judgment. Ordinary defects should enter net benefit as incremental expected loss; major defects require separate quality thresholds that cannot be crossed. Changes in sample size and detection power also matter, because distorted proportional denominators affect the accuracy of conclusions.

3. First-Pass CI Success Rate

This is a proxy metric for the integration smoothness of AI-generated code. Using a single pull request (PR) as the unit of analysis, it is the share of AI-assisted changes that pass required checks, such as Linter, Unit Tests, and Build, on their first push. The numerator is the number of PRs that pass all checks on the first attempt; the denominator is the total number of PRs.

  • How to measure: Track the proportion of analysis units that trigger and pass automated tests on the first attempt, as well as the average number of subsequent fixes required. Also record confounding factors such as flaky tests, infrastructure anomalies, and changes to checking rules. When comparing AI and non-AI changes, matching on the same work type, size, and risk strata is only confounding adjustment. Supporting causal inference requires random assignment or a rigorous quasi-experimental design. If that is not practical, teams should state clearly that they can perform descriptive monitoring only, observing concurrent changes rather than estimating the causal ROI of AI adoption from them.
  • How to observe: If this metric falls below baseline, it usually indicates that generated code conflicts with the current system architecture, or that the CI environment itself has areas worth optimizing. There is no absolute linear relationship between LOC and integration smoothness, so LOC cannot serve as an inverse metric.

Boundaries and Risk: When Metric Marginal Benefits Decline

Every measurement standard has an applicability boundary. Metric frameworks centered on output volume are especially likely to see declining marginal benefit in the following situations:

  1. Complex logic development: When work involves core business logic or algorithmic optimization, LOC and task difficulty do not have a stable monotonic relationship. Resolving a difficult architectural bottleneck may require changing only a dozen lines, so output volume cannot independently or reliably reflect delivery value.
  2. System refactoring and maintenance: Refactoring improves internal structure while preserving external behavior. Its result may increase or reduce LOC. If output volume is the metric, system incentives will tend toward retaining existing code rather than consolidating it. Refactoring should be evaluated by maintainability, coupling, and reduced risk, not simply by the number of lines added or removed.
  3. Cross-boundary integration: When AI-generated code needs to interact with external systems, the complexity of interface definitions and error handling is much higher than for internal logic, and validation cost rises materially.

Measurement standards need to match the nature of the work. For repetitive, pattern-based tasks, output volume can still serve as a supporting metric. For creative and complex tasks, outcome-oriented metrics and control of friction costs matter more.

A common hypothesis about the evolution of LLMs is that their generated code is already highly correct syntactically, but may still contain unnoticed edge cases without deep logical validation. If this trend holds in a specific team and workflow, then for those tasks, the focus of review may be shifting from syntactic correctness to logical completeness and architectural alignment.

This is an engineering challenge because checking logical completeness requires deeper domain knowledge. Under these conditions, an engineer’s role in these tasks may expand from code writer to system architect and quality gatekeeper. Precise implementation remains a solid foundation, while architectural judgment and quality validation become additional responsibilities.

Adoption Sequence: Build a Contextual Evaluation Framework

Teams can take an incremental-improvement approach to this shift.

First, establish a baseline. Before broadly introducing AI assistance, record the system’s current review lead time, escaped-defect rate, and CI success rate. Without a baseline, later performance evaluation loses its objective reference point.

Next, clarify workflow boundaries within the system. Separate tasks into generation-intensive work, such as test scripts and API scaffolding, and thinking-intensive work, such as architecture design and core logic. For the former, output volume can be a supporting metric. For the latter, architectural quality carries far more weight than output volume. Start by observing friction cost in generation-intensive tasks.

Finally, adjust weights dynamically. Workflows in different systems have different characteristics, and no single metric covers every situation. Dynamic weighting can be used only for decision prioritization, not for directly aggregating results across units. To build a composite score, first define the normalization method, source of weights, and sensitivity analysis. Otherwise, present metrics separately. Include review friction in the regular measurement cycle, cross-validating quantitative data with the team’s qualitative feedback from the review process.

Extending Systems Thinking

AI provides a new mechanism for teams to achieve engineering goals with a different cost structure. When attention shifts from output volume to the efficiency of handling demand, teams can use their own baselines and the three dimensions of review, defects, and CI to validate whether productivity has materially improved. In specific workflows, understanding system boundaries and controlling risk may be mechanisms that influence the degree of improvement, but this does not establish a universal causal relationship.

The market signal is not fully clear yet. Rather than maximizing output metrics under pressure to prove value quickly, begin by establishing a measurement baseline for your own system. Where is your team’s marginal time flowing: code generation, or requirements understanding and architectural confirmation?

This is a direction worth considering. What kind of measurement system do you want to build? Let the data speak, then make your own judgment.

Sources