Holding it right
Async Digital Ltd Cardiff, UK
Two talks from the AI Engineer conference seem to disagree about agent harnesses. Tejas Kumar of IBM took GPT-3.5 Turbo and made it do a real job by writing code around it, without changing the prompt. Dex Horthy of HumanLayer argues that no harness fixes what agent-run software factories do to a codebase, because models are trained to pass tests and nothing in training scores design. I stopped reading every line of my own code a while ago, which Horthy says you still have to do for anything that isn't small. So I sorted the checks around my work by what does the checking, and looked at two that went quietly blind. My answer is that the talks meet at a single line, what code can verify. My own record supports Horthy's account of the mechanism. On what to do about it, I have come to a different view.
A weak model with code around it
In his talk titled "Harnesses in AI: A Deep Dive", Kumar built an agent live on stage. Its job was to upvote the top story on Hacker News, and he gave it GPT-3.5 Turbo on purpose.
The first run reached the login page, gave up, and reported success anyway. He did not touch the prompt. He added plain code around the model instead, one piece at a time, including a check that the upvote really happened, until the final run worked. Here is .
First a step limit and a context trimmer. Then a function that read the agent's own tool history and refused to accept an upvote that never happened. That run still failed, but it stopped lying. Last came a login handler that spotted the login page and filled in the credentials itself, so the model never saw them. The final run worked.
His definition of a harness is "everything around the model that gives it grounding in reality". Near the end he points to OpenRAG, an open-source project he says IBM builds.
Harness engineering is not enough
In his talk titled "Harness Engineering is not Enough: Why Software Factories Fail", Horthy takes on the "lights-off software factory", where agents write the code and nobody reads it. HumanLayer tried running agents this way themselves in July 2025, and he puts the point where agents start to struggle in a codebase at three to six months. His argument is that no amount of harness engineering fixes this, because it is a training problem. Models are trained to pass tests, and nothing in that training scores design.
His cure is to keep reading every line of anything that isn't small, and to spend the effort before the build on product review, architecture, program design, and the order of implementation.
Like Kumar, he points to his company's work near the end. The difference is that HumanLayer sells tooling for the planning-first workflow the talk recommends, and its platform demo walks through the same steps.
The best case for Horthy
Kumar's task was tiny and fully specified. One login, one click, and success is a single fact the code can look up. A real codebase has no such fact for "this design will survive the next six months of changes". Kumar's method needs a verify function, and for design nobody knows how to write one.
The obvious workaround is to have a model review the code. Horthy has an answer to that too. In his words, "if the model knew what good code looks like, it would probably write it in the first place." A reviewer trained the same way inherits the same blind spot.
That lands squarely on me. My about page says I don't read every line any more, because I built a system of agents to review the code instead. That is exactly the kind of reviewer he is sceptical of. I don't agree with him on this point, and §5 explains why.
Where the two talks meet
Kumar's harness works because code can check whether it succeeded. Either the story was upvoted or it wasn't. His verify function reads the tool history and gets a yes or a no.
Horthy is talking about something code can't check. Whether a design will hold up has no function that returns true.
So the talks don't contradict each other. Kumar shows what a harness is good at, and Horthy points at what it can't see. The useful question for anyone running agents is which side of that line each of their checks sits on.
What checks my work
The setup I work in is called Ordova, and it isn't for sale. I sorted its checks by what does the checking.
Some are plain code. .
When an agent edits a Swift file through its file tools, a hook runs SwiftLint and fixes what it can. Another hook stops to ask me before such an edit adds a comment that switches a lint rule off. Neither sees an edit made from a shell script. While I drafted this note, a safety plugin refused to run one of the agent's commands, because it could not analyse the inline Python script inside it. Checks like these are cheap and they never get tired. None of them can say whether a design is good.
Some are models, and this is where I would put the ceiling a little higher than Horthy does. .
An auditor agent reads Swift against my written conventions for architecture, resource loading, access control, and localisation. Project-lead agents review a ticket or a pull request against a product's scope when I call them. These come closest to judging design, and they are the reviewer from §3.
Checking is easier than producing, and a reviewer with written rules and one narrow question notices things the writer never looked at. But it is still a model, and I have no number for how often it is right.
The last check is me. .
The agent is told to ask before anything hard to undo or anything that leaves the machine, and to wait for a go. That is an instruction, not a hook, so it holds only while the model follows it. A short list of destructive commands, such as hard resets and force pushes, is blocked in settings. And every run ends with me deciding whether to believe what came back.
So Ordova is not a lights-off factory. It is not what Horthy prescribes either, since I don't read every line. It sits somewhere between the two, and that is the position his argument puts to the test.
Where the checks went blind
The better evidence is where my checks failed quietly.
Within a week in September, a scripted edit by an agent turned a note into four copies of itself, and the check meant to catch that passed. One of the guards added afterwards, the one my rules called the most important, could never fail on a real edit. Here is in full.
On 8 September an agent edited a note with a short Python script. An operator-precedence slip made the replacement text the whole document, and the note came out as four copies of itself. The script exited 0. The check meant to catch a bad edit looked for a formatting marker, and four copies pass that check as easily as one. An unrelated search caught it minutes before it was committed.
The fix was a small editing module that refuses to write unless the anchor text appears the expected number of times and the new text is not the whole document. For four days my written rules also named what they called a byte-count comparison as the guard that mattered most. On 12 September it turned out that comparison could never fail, because a check that ran before it already guaranteed the arithmetic. It fired zero times in 60,000 random trials. A wrong input built separately did make it fire, so the code worked. Nothing real could ever reach it. It would not have caught the four copies either.
Neither failure was caught by the guard that was there to catch it. The first was found by accident. The second was found by asking whether the check could fail at all. That question was already a rule in my working principles, written on 6 September, two days before the first failure. A check tells you nothing until you have planted the defect it exists to catch and watched it deny. Having the rule written down did not stop that guard going untested for four days.
Every check has a blind spot, and something outside the check has to cover it.
My two failures are Horthy's argument on a small scale. Models are trained to satisfy the check, and checks built with those models go blind the same way, one level up.
So, am I holding it wrong?
Partly. Here is the split.
Where I am holding it wrong. My checks go blind in the way his argument predicts, as §6 shows. The model reviewers I rely on are the kind he is sceptical of, and I have no number for how often they are right. And I can't show that my codebases are holding up, because nothing I run measures it.
Where I am not. Not reading every line is not the mistake in itself. The mistake is not knowing where my checks stop. Where I know they stop, I read.
Horthy doesn't say checks are useless. The talk makes three points in their favour:
- Review agents raise the floor.
- Small changes can still go straight to an agent.
- His company says it is building better verifiers for software quality.
His position is that no check lifts the ceiling, so anything that isn't small still needs a person to read every line of it.
That is where I disagree. A harness can turn some of what good design means into checks that run, such as a lint rule, a written convention, or a hook that asks first. That is Kumar's method pointed at code quality. Where those checks can't judge, they can still tell me where to look, so my reading goes to where the checks stop. He is also careful to say that he can't prove models haven't improved at maintaining code, because no good benchmark measures it.
His account of the mechanism holds, and my own record backs it. What I take from both talks is that holding it right means knowing where each check stops, and putting a person at that point. The harness decides where my attention goes. It doesn't stand in for it.
I have stakes here as well. I sell a book about working with AI agents, Mastering the AI Coworker, and a verdict that the harness is most of the answer suits it. Ordova is my own setup, and I would rather it came out well. I made the case for what the harness does in Coordination lives in the harness, not the prompt.
By Horthy's account I am running his experiment. I don't read every line, so I should be able to say whether my codebases are falling apart. I can't yet, and the honest reason is the one he gives. Nothing I run measures it. That is a check I need to build, and by my own rule it won't count until I have watched it fail.
Both talks were transcribed on my Mac with WhisperKit's small Whisper model, not taken from the platform's captions. Both quotations match those captions word for word, but neither was checked by ear.
Every sentence about what my setup does was checked on 15 September 2026 against the live harness settings, hook scripts, agent definitions, and my written instructions, at the precision they show. The two failures in §6 come from my own dated working rules.
I worked on this with an AI agent, which transcribed the talks, read the config, and drafted the text. The disagreement with Horthy was mine, and so were several of the revisions. Two review agents asked for others. The drafting agent argued back, and some of that argument is in §3 and §5.