adlk.ioThere's a pull request open on Franz right now that I've read three times. A bot opened it. The title starts with "chore", it touches two files, and the whole change is five lines in a package.json plus the lockfile that follows from them. Six libraries move up a version. I've read diffs like this for ten years, and this one reads completely fine.
The machine that checks these things for me refused it. Twice, a week apart, and both times it needed three seconds.
Reading was all I ever had time for
The bot is Dependabot. If you haven't met it: it watches the libraries your software is built on, and every time one of them publishes a new version it opens a pull request that swaps the old number for the new one. I switched it on for the Franz monorepo in June. It has opened 70 pull requests since. More than one a day, every day, because there's a ton of libraries underneath 19 apps, packages and services, and they never stop moving.
For ten years my way of dealing with updates like these was to read them, quite carefully even. It felt like diligence. It was the only check that fit into the day, and it proves exactly nothing. The diff says a version number changed. Whether Franz still starts with that new version inside it is a question the diff can't answer, and answering it properly means building the app and running it, which is a morning gone for a one-line change. I never had that morning. So I merged the ones that looked harmless and let the rest sit. There's one in the old Franz repo that has been sitting since June 2021.
Two a night, against the real app
Since late July, the nightly loop that tests my own code picks up two of these pull requests a night, oldest first. For each one it merges the change onto the current state of the codebase, installs everything, builds the shared packages, puts a real database underneath and then starts the built desktop app. Does it launch. Can you actually use it: sidebar, settings, adding a service. About three and a half minutes later it writes its answer as a comment on the pull request: safe to merge, or not recommended.
It never merges anything. I do that in the coffee shop, quite often in the few minutes before someone joins me at the table.
The cap of two is deliberate. Two verdicts I can trust are worth more to me than 20 I'd skim. And a pass has an expiry date: if a week goes by and the codebase underneath has moved, the machine re-checks its own old opinion, as a yes on top of last Tuesday's code doesn't count for much today.
Two things I want to say plainly before this sounds like an invention.
Running tests on a schedule is as old as CI. Every team I've ever worked with had a pipeline that builds a dependency bump and runs the unit tests on it. What that pipeline almost never does is start the finished product afterwards and use it, because doing that for every pull request is slow and way too flaky for a pipeline that has to answer in minutes. It's the check that matters most and it's the one that gets cut.
And there's no AI in any of this. The whole path is shell scripts. The model's fingerprints are one level down: the test suite that gives this check its teeth is the thing agents built and keep repairing, which was part two of this series. The scheduler was never the missing piece. A suite worth running was, and for ten years I didn't have one. The idea was always available to me. Building and maintaining the verification is what got cheap.
The break was two levels down
Back to the refused one. The night it came up in the queue, the install died after three seconds. Somewhere underneath those six bumped libraries sits another one, jsdom, and its new version demands a newer Node than the machine runs. Newer by one minor version. The install refuses to start, so nothing gets built and nothing gets tested, and the machine writes on the pull request: not recommended to merge until this is understood.
A week later the bot rebased its pull request and the machine checked it again. Three seconds. Same answer.
I could have read that diff every morning for a month and never caught it, because the line that broke it isn't in the diff. It's two levels down, in a requirement that only shows itself once you install the thing for real.
The refusal is also what makes the rest worth anything. Since late July the machine has said "safe to merge" 20 times, and I've stopped double-checking it, which only works because I've watched it say no. A checker that never says no teaches you nothing about its yes.
The loop doesn't keep what it learns yet
In part two I wrote that the difference between a job you automate and a loop that improves is that the loop keeps what it learns. By that measure the dependency side is still a job. It runs the same check every night, and every "no" lands on me as a fresh mystery. The comment on that refused PR ends with "until this is understood", and understanding it is still mine to do. The machine found the problem in three seconds, then handed the thinking straight back. Next month it would flag the same kind of clash again as if it had never seen one.
This is the part of my setup I'm currently arguing about with a model in the mornings, at the same table where the rest of the loop got argued into existence. I don't know yet what I'll build, and I'd rather say that than pretend there's a roadmap. But the directions are taking shape.
A "no" should get investigated before I ever see it. Whose fault is it: the library, the machine, or my own code. For the refused PR up there the whole answer is one sentence, Node needs to move up one minor version, and it's sitting in the install log where an agent can read it just as well as I did.
The sentence sounds harmless, and it's anything but. Node sits underneath everything in this codebase, and moving it even one notch can break something seven levels away in a completely different corner of the app. So the fix needs the full verification all over again, which is exactly the kind of retesting the nightly loop is there to absorb.
Whatever the investigation concludes should stay in the loop, the way one bad night in July became rules the testing side still runs every night. The next clash of the same kind should be recognised, not re-diagnosed from zero.
And someone should read what actually changed inside a dependency. Not my diff, the library's own: its changelog, the code it rewrote between those two version numbers. No person sustains that across 70 pull requests, me included. A model doesn't get bored, and it could tell me the one place where a change touches code I actually have.
The end state I want is that this work disappears. Small updates that pass arrive batched with their evidence attached, big ones arrive as a prepared migration, and the reading I did for ten years stops existing as a job. What never disappears is the last step. The machine prepares, I commit. That line hasn't moved in two posts and it's not moving here.
Maybe something better than the current loop comes out of these mornings. Maybe all I get is an investigator for failed installs. Either one is more than I had last year.
If your team has a pile like this
Nothing in this post needs a solo founder. Every team I've worked with had this pile, and it's the first thing I'd hand over, because it belongs to nobody. There's no role called "validates the dependency updates", so there's nobody whose work you're taking.
The rule that transfers: don't automate the reading, automate the verification. All of it. Your CI already builds the bump and runs the unit tests. The check that's missing almost everywhere is the one that starts the product afterwards and uses it, the way a customer would. A check that doesn't exercise the thing your customers touch is an opinion with a green checkmark on it.
The suite doing that exercising is written and repaired by agents, and an agent that's allowed to fix tests has an obvious way to cheat. The rules that stop mine are part six.
If you're a founder working out how much of this you can hand over, and where the line should sit, that's the work I do with teams. Let's talk.
