Skip to content
All Posts

I Audited My Own AI Coding Agent Harness (and the Biggest Win Was Measuring Things I Assumed)

By Ralph Luis Abejuela
  • AI
  • LLM
  • Agent Harnesses
  • Prompt Engineering
  • Open Source

I had a system prompt I was proud of and a set of guesses about why my coding sessions felt slow. I never counted anything. When I finally did, several of my confident opinions turned out to be wrong.

This post covers that audit and the ten extensions that came out of it. It covers what the data said and what I built. It also covers what I decided not to build. That list includes one feature which ran in production for days and produced a number nobody ever read.

Why I audited the harness

I wanted to know whether the harness or the model was limiting me.

The answer matters because the harness appears to be the larger factor. LangChain moved a coding agent from 30th place to 5th on Terminal Bench 2.0, from 52.8% to 66.5%, while holding the model fixed at gpt-5.2-codex. They changed only the system prompt, the tools, and the middleware.

If the harness decides that much of the result, then tuning it by feel is guessing at most of the product’s quality.

The baseline I did not want to see

I wrote scripts/harness-audit.ps1 to parse the session transcripts directly. No sampling.

Metric Value
Sessions / events 40 / 2,973
User turns 116
Tool calls 1,304
Tool errors 82 (6.3%)
Tool calls per assistant turn 1.15 mean, and 982 of 1,133 turns sent exactly one
Turns that batched more than one call 13.3%
Cache hit rate 93.6%
Tool result payload 5.73M chars, about 1.43M tokens
Largest context seen 303,437 tokens, and 38% of calls ran over 100k

Two numbers in that table set the whole agenda.

The first is 1.15 calls per turn. The harness could already run a batch of tool calls at the same time. The model almost never sent one, so about 1,000 turns each paid a full round trip to the model to collect a single observation.

The second is 1.43M tokens of tool output against 0.80M tokens of model output. My context was mostly material the model read, not thinking the model did.

What the data said

The tool list was working against me. defaultTools was ["read", "powershell", "edit", "write"]. The tools grep, find, and ls existed as commands but were never registered as tools. In 1,304 calls they were used zero times.

So every search became a PowerShell one-liner: 584 calls, 45% of all traffic, with an 8.7% error rate and a p90 of 29 seconds. A native grep returns matched lines. A Select-String pipeline returns them wrapped in formatting objects and ANSI codes. The fix was one line in a config file.

edit was failing on anchors it invented. 15 of 133 edits failed, or 12.0%, because the model wrote oldText from memory instead of copying it from the file. The harness rejects the whole call when one anchor misses. Each failure therefore cost three round trips: the rejected edit, the read the model then needed, and the retry.

The file was already on disk, and the harness was holding the path that just failed. So edit-anchor-recovery.ts now attaches the current file content to the error message. I measured this one end to end. The edit failure rate fell from 22.1% to 12.9%.

Prompt caching had no instrument at all. In every session I counted 110.6M cache-read tokens and zero cache-write tokens. Either the free-tier providers do not report cache creation, or caching was read-only. Either way I could not measure the most important number in the harness.

Head-of-line blocking made batching worse than it looked. The parallel path in pi-agent-core prepared every call in a batch before it started any of them. Three of my extensions hook that preparation step, so a batch of six paid six times the hook latency before call one began. Fixing batching without fixing this would have made my measurements misleading.

The four optimizations I declined

Declining is the hard part, and it is where the papers helped most.

A cache for read results. The idea is that agents re-read files, so cache results by path, modification time, and size. My first measurement said 25.1% of reads were repeats, which sounded like a large win.

I measured it again, and the first number was wrong. Exact repeats, meaning the same path and the same range, were 8 of 192 reads, or 4.2%. Only 6 of those 8 had no write in between, so a cache could serve 3.1% of all reads. The rest were re-reads at a different offset, or re-reads after the model’s own edit, which a cache must not serve anyway.

Six cacheable calls in 40 sessions does not justify overriding read, the tool I trust most. A mistake in offset handling or in the 50KB truncation cap would cost far more than the seconds saved.

Pre-emptive edit anchor repair. Edits failed 82 times out of 761, or 10.8%, and 63 of those failures had a stale oldText. I had two candidate fixes and I measured both against 58 stale failures in 38 sessions.

Attaching file content to the first edit of an unread path reached 2% of failures, because the assumption behind it was false. 45 of the 58 failures were on a file the session had just written itself. Twelve were on a file it had read in full. Exactly one was on a file it had never read. The model was misremembering its own files at 400k tokens of context, not editing files it had never seen.

Anchor normalization reached 16%, which is 9 round trips in total. It would also rewrite oldText before the call runs, so a wrong match edits the wrong region and reports success. Nine round trips is not worth that risk on the most consequential tool in the harness.

A nudge toward parallel tool calls. Built, then shipped off. Salesforce’s Wide & Deep work measures 3 parallel calls per turn at 68% on BrowseComp, with 35 to 40% less cost and wall-clock time. It also finds GPT-5-Medium with parallelism at 62.2%, beating GPT-5-High without it at 54.9%.

So I built batch-nudge.ts. After three turns in a row with a single tool call, it appends a short request to send independent calls together. I set that threshold from the real distribution, where 83.9% of turns sit inside a run of three or more.

Then the A/B came back flat. Mean batch size was 1.00 with the reminder on and off, on both models I had available. One run looked like an improvement, from 13 tool calls down to 9. That task told the agent to work one file at a time, so the run measured the instruction and not my nudge.

It ships off by default. The mechanism verifiably fires. The effect is unproven. A change with no measured benefit does not belong on by default. Mine also costs tokens on every request and risks confusing a small model. You can turn it on with /batch-nudge on.

Speculative pre-warming. Deferred. PASTE, “Act While Thinking” from Microsoft Research, pre-runs likely follow-up tools while the model is still generating. It reports up to 48.5% lower average end-to-end latency and 61.9% at p99. The stable call sequence that it depends on is measurable in my transcripts. It stays on the list, unbought.

The feature that ran and told me nothing

read-sieve.ts is the most useful failure in this whole project, and I only found it because I went looking.

read was the largest single source of context bloat. It averaged 14.9k characters per call, and 9 results ran past the 50KB limit that pi imposes. The sieve keeps the top of the file and splits the rest into blocks of lines. It keeps any block that looks like an error or a warning. Then it asks a small model whether the agent needs each remaining block.

That model is Jev, a decision model from TypeSafe AI. It is built for machine use, and it returns a typed decision with a calibrated probability instead of text. That is the part that matters here. The sieve can hide a block only when Jev puts P(needed) below 0.1, because the model reports its own confidence. A chat model cannot do that. The site reports 193.6x faster and 444.6x cheaper than an LLM on the same workflows.

The extension shipped in shadow mode: make every model call, record the decision, change nothing. Shadow mode exists so you can read the numbers before you switch it on. Then I moved on to other work.

Three weeks later I went back to check whether it had ever done anything.

  • It was running. It loaded in 48 sessions and registered its recall tool in every one.
  • Its gate opened often. I replayed its own eligibility check over every stored transcript. 89 of 606 read results passed the 3,000-token gate, which is 461,427 tokens, spread over 18 sessions that ran after it shipped. None came near the 20-call budget per session.
  • The credentials worked. I called the endpoint with the stored key and got HTTP 422, which is a complaint about the request body and means the key passed. I sent a deliberately fake key and got HTTP 401.
  • It saved nothing. There is no write call anywhere in the extension. The counters lived in memory, and the only way to read them was a /read-sieve command.
  • Nobody ran that command. Every trace of its output in all 95 transcripts is the extension’s own source code being read back into a session.

So it ran, it spent money and latency on model calls, and it threw away every verdict each time the process reloaded. The one number that would have justified turning it on was never recorded, and I cannot recover it now. It is still loaded, and it will do the same thing again.

The fix is one line, and it is the kind of line I should have written first. Write each shadow decision to a log file. A shadow mode that keeps its notes in memory is not a shadow mode. It is a feature that costs money and reports nothing. If you build a measurement, make it survive the process.

What the papers changed

Every non-obvious decision in this audit traces to a specific result.

Paper What it measures What I did
Wide & Deep (Salesforce) 3 calls per turn, 68% on BrowseComp, 35 to 40% less cost and wall clock Built batch-nudge, shipped it off, rebuilt the A/B around wall clock
Don’t Break the Cache 41 to 80% lower cost and 13 to 31% faster first token, over 500 sessions Split my gated docs into Rules and Reference, and put all dynamic content last
PASTE (Microsoft) 48.5% lower average latency from pre-running tools during generation Deferred it. The prerequisite was measurable on my own transcripts
Beyond Token Savings About 35,000 runs. Policies using one third the tokens took 20 to 80% longer Wrote latency-metrics.ts and stopped arguing from token counts
Governance Decay 1,323 episodes. Summarizers delete constraints and violations rise from 0% to 30%, up to 59% Pinned my model-choice rule against compaction

“Don’t Break the Cache” earned a special mention. Its no-cache baseline is a UUID at the front of the system prompt, and it names timestamps as the classic mistake. I had a date stamp in my prompt. Its lesson is about placement. Dynamic content goes at the end of the prompt, so the cacheable part in front of it stays intact.

“Governance Decay” made me nervous, because my strictest rule is that only free models and DeepSeek models may ever be used. That rule governs nested runs and subagents, which you never see. If compaction drops it, a nested run quietly picks a paid model. So it now has two layers. The summarizer is told to copy the block into the summary, and after any compaction every request appends it again. The second layer is the guarantee. It stays inert otherwise. It never triggers compaction, and before any compaction it adds nothing, so the normal request is byte-for-byte identical and the cache is untouched.

The measurement that reframed the project

The audit argued its recommendations from token counts, and then it had to retract one of them.

Afterwards, Beyond Token Savings measured the premise directly over roughly 35,000 agent runs on SWE-bench Verified and Terminal-Bench 1.0. Policies that used about one third as many tokens took 20 to 80% longer than the uncompressed agent. Fewer tokens did not mean faster or cheaper. It also found that policies with equal success rates solved different tasks, and that the same policy behaved differently across models.

That removed the ground under my reasoning. The batch-nudge A/B had measured calls per turn, not wall clock. The 35 to 40% in Wide & Deep is a wall-clock and cost result. The real question was never whether the model batches more. It was whether the turn finishes faster.

So I wrote latency-metrics.ts. It records wall clock for each model request, each tool call, and each turn. It keeps three intervals apart: time to first byte, time to first token, and streaming time. A slow turn can then be traced to provider queueing, generation, or tool execution. It is observe-only. No handler returns a result, so it cannot change what it measures. Then I rewrote the batch-nudge A/B around wall clock, joining each injection against the latency rows. Each injection is a within-session comparison, so a change of model or provider load does not leak in.

That A/B still has not reached a conclusion. The instrumentation exists and its tests pass. I am not going to report a number I do not have.

What shipped

Ten extensions. Each one traces to a measurement, and each one has offline tests. The suite is 295 assertions in nine suites, all passing, in about 30 seconds.

Extension Problem it fixes
read-sieve Oversized read results. Shadow mode, and the verdict was never captured
context-economy Byte-identical tool results sent again on every turn
edit-anchor-recovery Rejected edits caused by anchors written from memory
docs-gate Holds the first call that touches a gated technology and sends the doc in the hold message
tool-choice Redirects raw HTTP and search-page scraping to the correct tools
mcp-arg-coerce Models that send "false" where the schema says boolean
batch-nudge Serial tool calling. Ships off, effect unproven
latency-metrics Wall clock, because tokens are not time
session-actions-brief Sends the session-action contract next to the tools it describes
agent-md-date Stops a stale date in the prompt from corrupting every time-sensitive answer

Three details I like, because in each case the plain option is the correct one.

The docs gate sends the document inside the hold message. The first version held the call and told the model to go read the file, which cost a whole extra turn. Sending the text as the tool result removes that turn. It does not weaken the rule, and the gate cannot be satisfied by a partial read, because there is no read to make partial.

Rules are enforced in code, not in prose. In one measured run an agent read the scraping instructions. Those instructions say to prefer the dedicated scraper, and the agent called Invoke-RestMethod anyway. Reading a rule does not make it binding. The blocks are bounded and they fail open, so a wrong pattern match can never lock a session. When a bound stops enforcing, the extension says so out loud instead of quietly stopping.

My system prompt went from 1,715 tokens to 960. That is a 56% cut. I decided it by mining 55 sessions and 3,244 tool calls to see which instructions had ever changed a decision. Prompt I never needed is prompt I pay for on every request forever.

What I would tell anyone doing this

  1. Measure before you change anything. My highest-confidence opinion before the audit was that read results should be cached. It would have been a waste.
  2. A decline with numbers is a result. “I built this and here is why I am not shipping it” is worth more than a feature nobody examined.
  3. A shadow mode that keeps its notes in memory is not a shadow mode. My most expensive mistake was an extension with no log file.
  4. Tokens are a proxy, and a bad one. If you are arguing about speed, measure wall clock.
  5. Check the source of your own numbers. Two of my eleven findings were retracted by my own later measurement. One claimed a bug that the extension had been written to fix, so the bug was history and not a gap.
  6. Enforce what the model keeps forgetting, and trust nothing it only reads.

Resources