# We're Automating the Wrong Part of Software Development

AI is making software cheaper to build. The harder problem, and the bigger opportunity, is understanding what to build in the first place.

We're getting remarkably good at building software.

AI can now write code, create interfaces, generate tests and, more and more, manage parts of the development process itself.

But something doesn't add up. If software has become so much easier to build, why aren't businesses suddenly producing much better software?

My suspicion is that we've been automating the easiest part of the problem. The difficult part was never only writing code. It was understanding what needed to be built in the first place.

## Software production is getting cheaper

The progress is real. Tools like Claude and Replit can take a description and produce working software in minutes.

Coding agents now work through tasks on their own. They write the code, run the tests, read the failures and try again.

Some developers have pushed this further. A technique nicknamed the ["Ralph" loop](https://ghuntley.com/ralph/) feeds the same instructions to a coding agent over and over until the work is done. Each pass picks up where the last one left off.

What's telling is where its creator says to look when it goes wrong: "If Ralph is building the wrong thing completely, then your specifications may be incorrect."

The evidence on productivity is still mixed. In [one controlled study](https://arxiv.org/abs/2302.06590), developers using GitHub Copilot finished a well-defined task 55.8% faster.

In [another](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/), experienced developers working in their own large, mature codebases took 19% longer with early-2025 AI tools. The researchers have since said newer tools likely do better, but their data can't yet say by how much.

My reading is that AI does best when the task is well defined. The more context the work depends on, the harder it gets.

None of this means autonomous software engineering is solved. Architecture, security, integration, deployment and maintenance are still hard. But the bottleneck is moving.

> When the cost of producing software falls, the quality of the instructions becomes more important.

## The real bottleneck is human understanding

People are experts at their jobs. That doesn't make them experts at explaining how their jobs work.

The philosopher Michael Polanyi put it in one line: ["we can know more than we can tell"](https://press.uchicago.edu/ucp/books/book/chicago/T/bo6035368.html). He called it tacit knowledge.

Experienced people make many decisions without noticing them. They recognise exceptions, apply judgement and read context.

When asked to describe their work, they tend to leave out what has become second nature. Those are often the details that matter most.

Now think about how software usually gets specified. A person describes their work to a consultant. The consultant interprets it and writes requirements. The requirements become specifications, and the specifications become software.

Each handover is a chance for something to be lost. Even when everyone means well, the result can be software that faithfully implements an incomplete picture of the business.

This isn't a new problem. In a [survey of 228 companies across 10 countries](https://arxiv.org/abs/1611.10288), "incomplete and/or hidden requirements" was the most cited requirements problem. It was also among the problems most often linked to project failure.

> The problem isn't that people don't understand their jobs. It's that they understand them so well, they struggle to explain everything they do.

## Building the wrong things correctly

Two examples show how this plays out.

**The website that was waiting for content.** A business asks for a sophisticated blog and thought-leadership section on its new website. The CEO has agreed to write. The agency designs a beautiful editorial experience.

The site launches, and the articles never arrive.

The software works. The business outcome doesn't happen. The brief described an aspiration as though it were an existing capability. A better briefing process would have asked who writes, how often, and who makes sure it happens.

The lesson: requirements need to separate what a business can do today from what it hopes to do.

**The quarterly invoicing problem.** A business invoices quarterly because invoicing more often takes too much admin. An analyst records quarterly invoicing as a requirement. The developers build software that supports it.

But quarterly invoicing was never the requirement. It was a workaround for a constraint. If software removed the constraint, the business might prefer to invoice monthly, or as each job is completed.

The lesson: requirements need to separate the underlying objective from the workarounds built around it.

In both cases, nobody made a mistake on purpose. Clients describe what they know. Consultants get enthusiastic about the thing they've been asked to build. Each side can reinforce the other's assumptions, and nobody questions the problem underneath.

> If we automate a process without understanding why it exists, we risk making its inefficiencies permanent.

## The opportunity is a better briefing process

AI's most interesting contribution may not be writing code. It may be helping people investigate, clarify and improve their own understanding of their work.

Imagine a briefing system that works as a conversation. It accepts incomplete descriptions, ordinary language and whatever documents already exist. Then, question by question, it builds up a picture of:

- the people, systems and information involved
- the current process and the objective behind it
- missing steps, exceptions and assumptions
- workarounds and the constraints that created them
- the difference between how things work now and how they should work
- commercial, legal, technical and operational risks
- the decisions that need explicit human approval

It asks targeted questions rather than burying people in technical detail. Standard technical decisions are handled with sensible defaults and established patterns.

One principle matters above the rest. The system shouldn't make commercial decisions on the customer's behalf. It should surface risks, alternatives and consequences, so the customer decides with better information.

I've written before about [test-driven briefing](/blog/you-suck-at-briefing-but-that-s-our-fault/), where every request becomes something that can be checked. This is the same idea, extended to how a whole business works.

![Two flows side by side. Left, where most of the investment is going: human brief, specifications, AI coding, software. Right, where the bigger opportunity may be: human understanding and iterative clarification repeat until the need is clear, then validated specifications, software factory, software.](/images/blog/automating-the-wrong-part-1.svg)

> The innovation isn't an AI that knows what your business should do. It's a system that helps you understand what your business is trying to achieve.

## Small improvements that compound

Large software transformations are risky. They take years, cost a lot, and can deliver something the business has outgrown by launch day.

There is another path. Take an employee who copies information from one system to another every day. Instead of replacing both systems, a software factory produces a small application that connects them.

The employee saves time immediately. The business gains a useful application. It also learns how its systems and information fit together, and that informs the next improvement.

Do that repeatedly and the applications start to add up. Think Voltron: each piece is useful on its own, and together they form something far more capable.

That doesn't happen by accident. Generating lots of small applications can create a new tangle instead of a platform. The applications need to share information structures, standards and interfaces from the start, and that takes deliberate architecture.

![Four small applications, each useful on its own, connect to a shared layer of information, standards and interfaces. Over time, the shared layer turns them into one connected platform.](/images/blog/automating-the-wrong-part-2.svg)

> Every piece of software should solve a problem today and contribute to something bigger tomorrow.

## Businesses need software before they need autonomous AI

Many businesses are being encouraged to bring in AI agents. Plenty still run on systems that don't talk to each other, data that doesn't match, information re-keyed by hand and processes that live in people's heads.

Putting AI on top of that tends to amplify the problems rather than fix them.

Fixing them isn't only preparation for AI, though. Automating a process, standardising information and connecting systems pays off now.

It also builds, gradually, what I think of as an organisational brain. That's a connected picture of the business's customers, jobs, processes, relationships and information. It's close to what I've called [business as code](/blog/business-as-code/).

Over time, that picture could give AI reliable context for spotting opportunities and supporting decisions. Clean data alone won't make autonomous AI reliable. Governance, testing, access controls and human oversight still matter. But context is where it starts.

> Don't reorganise your business around AI. Build better software around your business, and you'll create the foundations for AI to become useful.

## Every business could have its own software factory

Here's where this could lead.

An employee spots an operational problem and explains it in ordinary language. The system investigates the process and clarifies the requirement. It flags the risks and the decisions that need approval, and a person makes those decisions.

The result is a validated specification. A software production system generates, tests and delivers the application. The employee uses it, gives feedback and spots the next improvement.

Each cycle produces two things: useful software, and a better understanding of how the business works.

![A six-step loop: an employee spots a problem, the system clarifies it, the employee decides, software is built and tested, the improvement is measured, and the next opportunity is found. Every cycle produces useful software and a better understanding of the business.](/images/blog/automating-the-wrong-part-3.svg)

Over time, the business builds a software platform that reflects its own processes. AI models will keep changing. The operational knowledge and the software stay valuable.

> We're not trying to turn employees into software engineers. We're trying to give them the capabilities of software engineers, without taking them away from what they're already good at.

## Understanding comes first

The industry is investing heavily in making AI better at producing software. That's valuable work, and it will continue.

But software is only valuable when it serves a real business purpose. The bigger opportunity may be systems that help businesses understand their own work, find worthwhile improvements and turn them into software, one step at a time.

That could change the relationship between businesses and technology. Instead of buying tools and adapting to them, businesses could increasingly build software that reflects how they work.

The result isn't necessarily more AI inside businesses. It's businesses becoming more capable of improving themselves.

The future of software isn't only that anyone will be able to build it. It's that businesses will be able to solve problems that once needed expertise, resources and budgets they didn't have.

The most valuable technology may not be the system that writes the code. It may be the system that helps people understand what needs to be built.

I don't think that's solved yet. So here's the question I'd put to anyone working on it. Can we create a reliable, scalable way to capture business understanding and turn it into continuously improving software, without a specialist consultant at every step?

If you're thinking about the same problem, I'd like to hear from you.

## Sources

- [Ralph Wiggum as a "software engineer"](https://ghuntley.com/ralph/) — Geoffrey Huntley, 2025
- [The Impact of AI on Developer Productivity: Evidence from GitHub Copilot](https://arxiv.org/abs/2302.06590) — Peng, Kalliamvakou, Cihon and Demirer, 2023
- [Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) — METR, 2025
- [The Tacit Dimension](https://press.uchicago.edu/ucp/books/book/chicago/T/bo6035368.html) — Michael Polanyi, University of Chicago Press (first published 1966)
- [Naming the Pain in Requirements Engineering: Contemporary Problems, Causes, and Effects in Practice](https://arxiv.org/abs/1611.10288) — Méndez Fernández, Wagner et al., Empirical Software Engineering