adlk.ioI was in the coffee shop by half past eight and the overnight summary was already on my phone. 194 passed, 74 failed.
Then the list. Onboarding. Sign-up. Services. Settings. The trial offer. 24 files, most of the product, all red.
The report was telling me something major was going on. Nothing was actually wrong with the app.
A Postgres update had installed itself on that machine during the night and changed the way it hands out permissions. Every test that touches the database failed, and almost every test touches the database. The app was fine. The floor it was standing on wasn't.
Not one of those 74 tests could tell me that. Each only sees what happens in front of it, a button that never appeared, a page that never loaded. The layer underneath knew exactly what was wrong and said so 418 times that night, into a file nobody was reading.
The part that actually bothered me
51 minutes of machine time went into a run that could never have passed. That's the cheap part. I was asleep.
Here's the expensive part. When that suite goes red, the next thing in the queue is an agent whose whole job is to make failing tests pass. It's quite good at this.
Hand a person that pile of failures and the reply comes back with a hedge in it. Everything is failing at once, which seems unlikely, so it might be something on my end. Before you tell the room the building is on fire, you check whether it's your own smoke alarm. An agent doesn't send that message. It says yes and starts work.
It didn't start work that night, and the only reason is a number. There's a cap: more than five failing specs and it stays out, because five failures at once are usually one cause, and patching them one at a time papers over something bigger.
74 is more than five.
So what saved me was the size of the breakage. I set that cap to catch systemic regressions in the code. It was never meant to protect me from a broken machine, and it only managed to because this machine broke completely.
Imagine the update had taken out a corner of it instead of all of it. Three or four tests fail. The cap waves it straight through, and an agent spends the next four hours working on a database that is refusing every connection it receives.
A stupid gate in front of a very capable worker
Two things changed the next morning. The first is almost embarrassing: before anything runs, something now checks whether it can run at all.
The catch is where you ask from. That morning I checked the database by hand and it answered me perfectly. Of course it did. I wasn't standing where the tests were standing, and they don't reach it the way I do. The check now asks the same question the way they have to.
The second change matters more, and it has nothing much to do with tests. The night now has a third possible outcome: "I couldn't run." Before there were two, passed and failed, and if you only give something two answers it will hand you one of them even when neither is true.
Deciding which of the three it is takes no intelligence at all, and I want to keep it that way. The thing waiting behind that gate is clever, tireless and willing to help. None of those are qualities you want in whatever decides to wake it.
What the agent is actually for
None of this is why the AI is there. Running tests on a schedule is a solved problem and has been for 20 years. A timer can do it. A timer has never once told me what's wrong.
Here's a night where the suite really did fail. The verdict lands six minutes past two. The agent starts nine minutes after that. By half past there's a pull request open with a per-failure explanation in it, and I'm asleep for all of it.
What it does in those 18 minutes is the work I never had time for. It reproduces the failure. It decides, for each one, whether the app is broken or the test is broken, which is the judgement call that used to eat my morning and the one thing no amount of scripting ever did for me. Then it fixes whichever it was, runs it again, and writes down what it concluded.
Both times it ran for real it came back with the same answer: this is the test's fault, not the app's. And both times it added something I didn't expect. It couldn't fully verify the fix, so it said so, on the first line of the pull request, in capitals. It's required to. A fix it couldn't run isn't a fix, and it isn't allowed to let me read it as one.
So the machine can hedge after all. It just has to be made to, which is most of part four of this series.
And it stops there. It never merges. That line hasn't moved since part one.
The loop keeps what it learns
This is the thing I was trying to describe in part one and couldn't do properly without an example.
If I did this once a quarter, that night would have cost me a morning and taught me nothing. I'd have fixed the permissions, sworn a bit and moved on, and the next time a machine lied to me I'd have started from zero again.
Instead the night is still in there. It changed three things, and all three are still running tonight.
The third one is my favourite. The output that thing watches out for is the literal text the database was printing at two in the morning, copied straight out of the log, so that particular lie can never work twice.
Which is the difference between a job you automate and a loop that improves. One does the same thing every night. The other gets slightly harder to fool each time somebody fools it.
Here's every night since, including the ones I'm not proud of.
Setting this up was quite complicated and full of false alarms. I still have to turn a few screws from time to time. But the green nights are boring and the red nights are worth reading, which is what I wanted and didn't have.
Anything you hand over needs a way to say "it wasn't me"
If you're pointing an agent at anything, this is the bit I'd steal.
Work out what happens on the day it can't do the job at all, and make sure that looks different from the day the job goes badly. Those are different sentences and people say them differently without being taught. A new colleague says "I couldn't get in, is my access set up yet?" A system says FAILED and waits for you to work it out.
It matters more with an agent than it does with a person, for one reason.
Someone who thinks the question is wrong pushes back before they start. You've had that for free your whole career, from every colleague you've ever briefed badly, and it's the first thing you stop getting when you hand the work to a machine.
It's the first of several things I've had to teach mine to doubt. The most uncomfortable one is part six, where a marketing agent grades her own predictions every week and publishes a hit rate she doesn't come out of well.
If you're a founder working out how much of this you can hand over, and where the line should sit, that's the work I do with teams. Let's talk.
