every week there’s a new paper promising self-improving agents. skill libraries. episodic memories. error-correction graphs. i read a lot of them, because KAOS is exactly the kind of harness they’re supposed to plug into. and for months i kept hitting the same wall: the paper shows +20 points on a benchmark, i spend a cycle building it in, and it does… nothing. or worse.
so we stopped guessing. since v0.7, KAOS has one rule: no new mechanism ships until it survives a kill test that was written and sha256-locked before the mechanism’s code existed. we wrote up that machinery in our first paper, Plasticity Without Parameters. this post is about what the machinery caught — and the new paper it turned into.
the mechanism that aced everything
last week we evaluated a published error-correction idea: convert agent trajectories into little graphs, diff a failed run against a similar successful one, and the diff points at the exact step where things went wrong. clever. training-free. no embeddings. it fit our constraints, so it earned a probe.
per the discipline, i froze four kill gates before writing a line of feature code: beat our existing failure-localizer by 10 points, survive a wrong-pair lesion, pass a pairing-precision bar — and one reality gate: real trajectories in our live database had to actually have the repeated-action structure the whole idea depends on.
the results came back looking like a victory lap:
- 1.000 localization accuracy — against our incumbent’s 0.188 (which, fun fact, scored below random on this slice; the blind spot it was accused of is real)
- every lesion and precision gate: passed
- the reality gate: passed at 10× the required floor
verdict, computed by the frozen gates: ACCEPT. and then i looked closer at that last number, because a 10× margin on the one gate i expected to fail is not a win — it’s a smell.
the vacuous pass
the reality gate asked: do real agents revisit the same normalized action often enough for a graph diff to grip? median “node reuse” had to clear 1.30. it came back 10.0. here’s why: 86% of the tool calls in our live database come from search workers that call one tool in a loop. their trajectories normalize to a single repeated label. one label, repeated ten times, is technically “reuse of 10.0” — and it is exactly as useless for diffing as a trajectory where every step is unique. the gate was written to catch one kind of degenerate trajectory and got satisfied by the opposite kind.
among the twelve agents whose trajectories were actually diverse enough to diff? median reuse 1.000. zero of twelve reached the floor. the property this mechanism depends on does not exist in our real workload.
so: the verdict stays ACCEPT — the gates are frozen, and we don’t retune after seeing results, ever. but the disposition is do-not-ship, and there’s now a regression test that fails the build if this mechanism ever enters the shipped package. (this verdict/disposition split, plus a rule that audits can only ever downgrade a disposition — never rescue one — is now codified in the discipline doc.)
thirteen candidates, one pattern
here’s the thing: this wasn’t a one-off. it was the third time a mechanism died this exact way. an action-validation layer once returned VOID because our organic workload had produced two qualifying incidents against a pre-registered floor of 200. a memory-consolidation family got rejected because lexical-only retrieval can’t simultaneously keep exact tokens and bridge to new queries. and now a trajectory-graph mechanism died on argument entropy.
none of them died because their ideas were wrong. they died because our workload was missing something their benchmark had quietly provided for free.
the new paper names these freebies benchmark gifts, and identifies four — each measurable on your existing telemetry with read-only queries:
| gift | what the benchmark provides | our workload, measured |
|---|---|---|
| closed action vocabulary | actions normalize to a small fixed set, so patterns repeat | ABSENT — 89% of substantive trajectories collapse to one repeated label; the diffable middle: 0/12 |
| dense outcome signals | a reward every episode, often every step | ABSENT — 2 of 1,639 tool calls errored; outcome truth exists only at episode grain |
| lexically anchored task text | templated task strings, trivial retrieval | PRESENT — 85% of real task texts carry paths, identifiers, error codes |
| per-task expert references | a known-good trajectory for every task | ABSENT — exact failed→success pairs: 6.8% |
and the ledger lines up with the axes. the mechanisms that died each consumed an absent gift. the one family that survived measurement and ships in KAOS today — plain BM25 retrieval re-ranked by usage statistics — consumes the one gift our workload actually supplies: hard lexical anchors. real tasks name their files.
the honest part: the axes were formulated after watching these mechanisms die. on our own ledger this is retrodiction, not prediction — the paper says so in bold, because a theory that can’t be embarrassed is worthless. the falsifiable bet is about your workload: measure the four axes, adopt a mechanism, see if its fate tracks them. we published the axes to be shot at.
the two papers, and which to read first
this became our second paper, and the pair now has a deliberate division of labor:
- Plasticity Without Parameters is the machine — the harness’s self-tuning memory and the kill-test discipline that gates every change to it. read it first if you want to trust the instrument.
- Benchmark Gifts is the field report — thirteen candidates through the machine, what each died of, and the afternoon-cheap workload audit that falls out. read it first if you want to know what the instrument caught.
every number in both traces to a committed artifact in the repo — locks, verdicts, results files, the works. that includes the embarrassing ones.
the takeaway, if you build on agents
before you spend an engineering cycle on the next self-improvement paper, spend an afternoon on four queries against your own logs: how entropic are your tool arguments? where does outcome truth actually live? do your task descriptions name their objects? do failed tasks have successful twins? then read the paper’s evaluation section and list what its benchmark handed it for free. if the mechanism’s favorite gift is one your workload doesn’t give — an 81-point benchmark win will not save it.
the simplest onboarding i can offer, as always: open Claude Code in whatever project you’re already working on and say
use github.com/canivel/kaos on this project— danilo