Measure Disk 16z9

The illusion of AI productivity in software development and what to measure instead

Building quality controls

At Pearl, we’ve experienced similar aforementioned challenges when using AI to automate parts of our software development process. For example, we asked AI to help create pipeline automation, but it generated GitHub Actions workflows for an engineering environment that uses Azure DevOps and Jenkins. Our astute engineers caught the mismatch during review and had to rewrite the integration.

The code was plausible, but it didn’t account for our actual infrastructure. This is a clear example of how AI-generated code can be convincing despite lacking important context. To catch these instances before advancing too far in the development process, we have a quality-control process that combines AI-assisted review with human oversight based on risk. Lower-risk changes, such as unit tests, can often be reviewed primarily by AI, while critical production-facing code still receives human review. Our AI reviews identify an average of 16 errors per 10 pull requests, including security and architectural issues. The goal is not to remove humans from the review process, but to use automation to catch issues efficiently and reserve human judgment for decisions where risk and complexity warrant it.

Measuring productivity across the entire development lifecycle

Once appropriate quality controls are in place, CIOs need to assess whether AI is delivering measurable productivity gains. AI adoption metrics (token usage, developer participation, etc.) show how much a company is using AI, but not necessarily whether it is becoming more productive. A more useful approach is to track a combination of delivery, effort and quality measures:

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *