Software Engineering in the Age of AI

“Build X. Make no mistakes.” That’s a real prompt, and whole apps are being built from it. It’s the most transformative thing to happen to programming in my career, and it fills me with dread.

From talking to friends and colleagues, I might be in the minority, but I feel we’re heading in a strange direction. Most of the time, I think we’re using AI wrongly, and I’m guilty of it too.

To explain why, let me take you back to my school days. Computer Science was one of our subjects, though not a serious one; it felt more like a fancy extra-curricular. There wasn’t much ‘science’ in it. It was mostly theory, an introduction to what computers could do. We learnt about input and output devices, storage, networks and the internet, and somehow even algorithm flowcharts. Nothing fancy: adding two numbers, drawn as a flowchart. That sort of thing.

One chapter left an impression, though: Programming Languages. Looking back, it was a follow-up to the flowcharts: these were the languages you could use to write one down. Languages computers understand, so you can talk to them and make them do the things your flowchart described. In it was a section called Generations of Programming Languages, a peek into their history. In short, it split them into five generations:

1GL, low level: binary machine code. The language the computer understands directly.
2GL, low level: assembly. One step above binary, so humans can write instructions as words.
3GL, high level: FORTRAN, COBOL, Pascal, C and the languages that followed it (C++, Java). Words combine into statements, an abstraction over assembly.
4GL, high level: SQL, MATLAB. Close to English sentences or mathematical expressions.
5GL, high level: AI. Plain English sentences, an abstraction over everything below. Still to come.

That’s my textbook’s version, anyway. The real 5GL of the 1980s was about describing a problem and letting the machine find the solution, and it was meant to be the future too, until it turned out that working out how was still the hard part.

Each generation makes it easier to tell the computer what to do. Even back then, this moment was anticipated. And the further down the generations you go, the more skill, rigour and expertise it takes. Ask software engineers today whether they could build an app in assembly or binary, and I’d guess 90% of us (me included) couldn’t. I dabbled with assembly for K8, my CHIP-8 emulator, and it’s hard. I can work in the third, fourth and fifth generations.

For better or worse, this reveals a gap: not in the language, but in the thinking of the person telling the computer what to do. What matters isn’t only the language; it’s systems thinking, engineering thinking. However deep your knowledge of a language’s low-level details, it’s systems thinking that separates an engineer from someone who just knows syntax. Fred Brooks put it this way in No Silver Bullet, back in 1986:

I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation. We still make syntax errors, to be sure; but they are fuzz compared with the conceptual errors in most systems.

In 5GL, the syntax is simply natural language. If you can speak, you can build something for yourself. You can ask the computer directly instead of relying on someone else’s software.

But can you build things for others? That’s a different game altogether, and I don’t think you can, not yet. Not large-scale software used by people across the world. That still takes systems thinking. You can learn it, of course, without knowing the syntax of any language below 5GL. But will everyone?

There’s a reason it’s called engineering: it’s a specialist skill, knowing how to think in a way that scales. Many engineers have spent countless hours learning it, often without realising. It can be learnt, but not really taught. It comes from experience: building things, watching them break, fixing them, re-architecting, and carrying the lessons into the next thing.

I saw this recently at work. We deliver a machine learning model, from a cloud service, to a feature used by hundreds of millions of people, and the delivery mechanism was built with AI. The code worked. Then the service started throttling us. When I dug in, I found the gap: the way the code asked the service for the model, every request also checked the network for a newer version. On one phone, that’s nothing. Across all of them, every time the model is needed, it’s a flood.

The first fix was to go to the network only when there was no model on disk. Better, but it only moved the problem: whenever the model was needed and missing, the app still went to the network, and at that scale it still throttles. What it needed was a different shape, not a patch.

So I took it on and moved it onto the pattern our other model downloads already used: only one background worker ever makes network calls. The code that serves the model only reads from disk; if there’s no model yet, it says so and moves on. Downloading and serving became two separate jobs, with a clear boundary between them.

None of that is syntax. The AI wrote good code for the problem it was given. What it couldn’t see was the system: hundreds of millions of devices, a shared quota, a serving path that should never have touched the network in the first place, and a pattern we’d already settled on for exactly this.

Even the follow-up proved it. To test the worker, I built a QA menu to run it by hand. When I asked AI to show each run’s result on screen, it told me, twice, that it wasn’t possible. The worker already hands back its result; I had to point it there myself.

The AI’s limits weren’t the real problem in that story. Limits are fine if you know them. The problem is how lazy AI lets us be. When working code arrives in seconds, it’s tempting to take it and move on without asking how it behaves on hundreds of millions of phones. I catch myself doing it too.

So people who keep thinking for themselves, even in the age of AI, will thrive. They can build software that scales, and that’s what software engineering is about.

And there’s plenty left to build. We need harnesses around how AI writes software, to make it more deterministic and less probabilistic, and that means setting boundaries on the system: its inputs and its outputs. It’s already happening, with something new every other day: RAG, MCP, skills, loop engineering, agentic engineering. Some of it will turn out to be the right direction; some won’t. All of it is systems thinking.

What worries me is the narrative being built around coding with AI. It’s marketed as a replacement for software engineers, and I don’t think it is. It’s on its way to becoming 5GL. But because of that narrative, software engineers are living in fear. They’re abandoning systems thinking, because it takes time, and according to the same marketing, you’re not supposed to take time to ship anything in the age of AI. That’s the root of my dread, and that’s what I push back on.

Systems thinking is the most important skill a software engineer has: thinking things through with your own brain, rather than handing off one more hard problem with a prompt that says “Make no mistakes.”

I wrote this with an AI as my editor. It tidied my language, suggested structure and drafted some passages from my notes. The ideas, the story and every decision about what stayed are mine. Which is rather the point.