Harness Engineering, Part 1: The Model Is Not the Agent

I have written about skills, context engineering and loop engineering as three separate topics. They are three corners of one thing, and that thing now has a name: the harness.

This series is the tutorial I wanted when I started. Three parts, one use case that runs through all of them, and every number traced to where it came from.

The same model, two different results

Here is the task I will use for the whole series. I had 12 statistics about AI agents that reached me secondhand, through posts quoting other posts. I wanted to know which ones hold up.

First attempt: paste the 12 claims into the model and ask which are true.

The answer was not a disaster. The model said up front that it had fetched nothing and was working from recall. It even flagged four claims as untraceable.

That is exactly the problem. Eight “true” verdicts with plausible URLs, and no way to tell a right row from a wrong one without redoing the work myself.

Second attempt: same model (Opus 5.5), same 12 claims, with a small harness around it. It took 34 turns, 115 seconds and $0.52, with a smaller model handling the web searches. Nine verdicts pointed to a page that was opened in that run. Three were marked “unresolved”: one source could not be found, and two sites blocked automated access. I ran a script over the output, a second session reviewed it, and I opened the two blocked pages by hand.

I will be straight about what that bought. All eight recalled “true” verdicts turned out to be right. The harness did not make the model smarter. What changed is that I could check the answer, and the checking found things:

• Three of the four claims the plain prompt gave up on had a real source.
• One of those was being misquoted. The post I got it from said 74% of agent loops state what counts as finished. The paper says 74% name their end states and only 44% define “done”.
• One statistic had no source anyone could find. I dropped it from this series.

Parts 2 and 3 build that harness piece by piece. This part explains what it is.

Agent = model + harness

A model turns text into text. That is all it does. It cannot open a file, run a command, or remember what it said one call ago.

Everything else is the harness: the software that builds what the model reads, runs the tools it asks for, stores what happened, and decides when the task is over.

If you work with data, you already know this split. The model is the query engine. The harness is the orchestrator around it: scheduling, retries, state, and the data quality check that says whether the output is fit to ship. Nobody blames Spark when the DAG has no validation step.

This is not a small effect. LangChain moved its coding agent from just outside the Top 30 to the Top 5 on Terminal Bench 2.0, from 52.8 to 66.5, by changing only the harness. The model stayed the same.

The seven parts

A source-code study of eleven production coding agents, from a 100-line research tool to Claude Code and Codex, found that every one of them takes a position on the same seven parts, even when the position is to leave one out. Only the size changes.

Seen from the user’s side, the list splits in two. Most of it lets the agent do the work, including a clear goal and a way to verify it. Three things exist so you can trust the result: permissions and limits, a record of what happened, and a way to measure whether a change made things better.

The loop, and where it stops

The loop is the same in every product. Send the task plus everything that has happened so far. The model answers with either a tool request or a final message. Run the tool, append the result, send it all again.

You cannot change much of that inner loop, and you do not need to. The same study found that a more sophisticated loop does not predict better results.

What you own is the ending. By default an agent is done when it says it is done. That sentence reports what the model believes, not what happened.

So write the ending first, and write it in things a script can count:

Notice what is missing: any adjective. “Until the research is thorough” is not a stop rule.

Three more decisions belong here:

1. The bad ending. A run that stops halfway either restarts with a note about what failed, or waits for you. If you do not pick, the agent picks.
2. Hard caps. Turns, minutes, or money. An agent on its fortieth pass at the same problem will not fix it on the forty-first.
3. A log. If it runs without you watching, write every turn to a file.

This costs one sentence, and it is often skipped. A study of fifty published agent loops found that 74% name their end states, but only 44% define what “done” means. (A single-author preprint, so treat it as a hint.)

The tools it can see

The model does not see your systems. It sees a menu of function descriptions, and that menu is its entire world.

Start small. The eleven-system study recommends beginning with a shell and adding a tool only when you watch a real failure. More tools is not more capability. Unless your harness loads them on demand, every tool description is sent on every turn, and two tools with similar names cost more than a missing one.

Make errors useful. “Invalid request” tells the model nothing, so it guesses. “No match for that text, but there is a similar line at 42 with different indentation” tells it what to do next.

Match the tools to the model. A controlled experiment swapped a set of typed tools for a bare shell and held everything else fixed:

The smallest model needed the scaffolding. The largest was slowed down by it. The third row shows it also depends on the task. The experiment ran each task once and used open-weight models, not Claude, so read it as a direction to test rather than a rule.

For the research brief I gave the agent seven tools and no shell. The tool list and the spending cap are flags on the command, which is the point: they are enforced by the harness, not requested in the prompt.

claude -p "Do the task defined in SPEC.md, following CLAUDE.md." \
  --tools "Read,Write,Edit,Glob,Grep,WebSearch,WebFetch" \
  --permission-mode dontAsk \
  --max-budget-usd 6

That last choice paid off in a way I did not plan. When the run finished, the agent’s report said it could not run the check script because it had no shell, so that check was “still owed”. It did not claim a pass it could not prove. A tool that was never offered cannot be misused, which is a stronger guarantee than an instruction asking nicely.

Try it

Four small tasks. None needs code.

1. Write down three instructions you have typed into an AI tool more than once. That is your undesigned harness.
2. Pick a task you repeat weekly. Write its stop rule. If it contains an adjective, replace it with a number.
3. Decide the bad ending for that task: restart with a note, or wait for you.
4. Count the tools and connections your agent has loaded. Remove one you have not used this month.

Resources

• Harness Engineering: explicado desde cero, santi. The clearest build-it-from-nothing explanation I have read. In Spanish.
• Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents, Barbaste et al. The eleven-system study behind the seven parts.
• An Empirical Study of Harness Design for Coding Agents, Fan et al. The typed tools versus shell experiment.
• How to Design an Agent Harness: six decisions, Yarchi. Source of the “two halves” framing.
• Stop Hand-Holding Your Coding Agent: Engineering the Loops that Replace Step-by-Step Prompting, Sandeco Macedo. The fifty-loop study.
• Improving Deep Agents with harness engineering, LangChain.

Part one of three. Next: what the model sees, what survives a crash, what the agent is allowed to touch, and who decides the work is done.