Is AI marking its own homework?

This week, PHP2Shell caused mayhem around the world.

PHP is the code behind roughly 70% of websites. On Wednesday its team released fixes for two critical flaws. The same day, security researchers were seeing them attacked in the wild.

The National Cyber Security Centre urged UK organisations to take immediate action, and council and police websites were reported offline while people scrambled to patch.

Our first thought was the obvious one. Is this interesting?

Then, about a second later… is this us?

Luckily, our platform wasn’t affected. Which left us free to get properly nosy about the interesting part.

Person or machine?

A bit of both, as it turns out.

The flaws were found by researchers working with AI, and they’ve been open about it. One security firm made AI the headline. The faulty code had been sitting in PHP for over a decade, and careful people had looked in that corner before.

But the machine didn’t do it alone. A person still had to point it at the right corner, prove the problem was real and report it. Then the people who look after PHP wrote the fixes, some of them working full time, then through weekends and around the clock. Hats off to them.

Never mark your own

Remember swapping exercise books with the kid next to you? There was one rule. You never marked your own.

That’s what makes this week so interesting. AI now writes an enormous amount of the world’s code. Google says 75% of its new code is AI-generated, up from 25% in 2024. And now AI is being sent in to check code too. Which sounds like the same pupil holding the same red pen.

Except that isn’t quite what happened here. The AI that helped find PHP2Shell didn’t write PHP. It was marking someone else’s homework (old homework, at that!) and spotted a mistake that had slipped past careful readers for years. Then people checked the marks.

That’s more like an examiner. A very patient one.

It isn’t a one-off

In February, Anthropic said its Claude model had found and validated more than 500 serious flaws in open-source software, some of them hiding for decades. The PHP Foundation has an AI Security Engineer in Residence. We’ve tested our own system with Fable, Anthropic’s most advanced model, and we’d recommend it. A second pair of eyes has never been easier to come by.

It isn’t all smooth. The maker of curl, a little tool inside a lot of software, closed his bug bounty, the cash reward for reporting flaws, after a flood of poor-quality reports. To be fair, he’s also acknowledged AI can be a fine bug-hunting aid.

And the red pen isn’t only in friendly hands. As the PHP Foundation put it, tools now find bugs faster than ever, “for defenders and attackers alike”.

So who marks the marker?

The same Foundation post has the best answer we’ve seen: “What tools cannot do is decide on the right fix…”

AI can read every line, all night, without getting bored. A person still decides what’s real and when the work is done.

Preferably one who’s had their coffee.