Six candidates evaluated. Zero shipped.
KAOS v0.9 makes the discipline the deliverable.

most frameworks release notes follow a pattern. new mechanism, demo numbers, ship. ship again next quarter. the field assumes that visible velocity means the framework is improving. v0.9 ships no new mechanism. not because the cupboard is bare — five candidates were evaluated this cycle — but because none of them earned it.

that sentence sounds defensive. it is not. it took roughly a month of work to confidently say it, and that month of work is what v0.9 actually ships: a primitive that makes "did the candidate earn it" a question with a binding, hash-locked, retune-proof answer.

The problem this release solves

through the v0.8 series KAOS absorbed several research-paper-backed mechanisms in quick succession: neuroplasticity from biological consolidation models, the EvoSkills surrogate-verifier loop, MemPalace's AAAK compaction, CORAL co-evolution, a critical-step localizer, the failure-taxonomy and ISA/ISC layers. each one came with a measured win on a benchmark, but the benchmarks were authored alongside the mechanism. that is fine when the framework is small. it stops being fine when you are deciding which of five competing ideas to integrate next.

the question is no longer "does this mechanism work in isolation" but "does this mechanism still help on top of everything KAOS already has". the v0.8.3 native baseline is strong — localizer + diagnoser + retry-with-feedback already absorbs a lot of what new mechanisms claim. and there is a published prior worth taking seriously: SWE-Skills-Bench reports that 39 of 49 deployed agent skills produced zero gain on their target workloads and 3 actively degraded performance. shipping a mechanism without a test that can kill it is shipping a regression with extra steps.

so for v0.9 we sat down and built the kill switch.

The kill switch, in one paragraph

kaos.eval.harness is a small module — six files, under 500 lines. it gives you a Probe base class with a fixed lifecycle: write the kill gates into a JSON manifest, take its sha256, add that hash to a pre-registered allow-list in code, build the arms (B0 baseline, FULL feature, L1/L2 lesions), let the harness execute them, send the per-query outcomes through a blind judge on an anonymised stream, and emit a verdict by uniform rule: ACCEPT iff every kill-gate passes, REJECT if any kill-gate fails, VOID if the sanity floor fails or the judge audit is below threshold. editing the manifest after results land changes its sha256 and the harness refuses to run. there is no retune-and-rerun affordance — that is the entire point.

the CLI surface is three commands:

kaos eval probe run     --probe pkg.module:ClassName --out-dir DIR
kaos eval probe verify  --probe pkg.module:ClassName --results PATH
kaos eval probe falsify --probe pkg.module:ClassName

falsify is the one that matters most. it substitutes the FULL arm with B0 (or whichever baseline you choose) and asserts the harness emits [KILL: G1]. a harness in which the feature cannot lose is inadmissible, and any later "pass" from it is meaningless. you run falsify before you trust any run. the synthesis-consolidation bench from v0.8.x already used this pattern by hand; v0.9 makes it a single command.

What the discipline did this cycle

the v0.9 research arc evaluated six mechanism candidates against KAOS's v0.8.3 native baseline. the table below is the actual ledger.

two REJECTs and a DO-NOT-BUILD are arguments from evidence. a VOID is an argument from honesty about evidence absence. the framework would have been worse if any of these had shipped under the older "looks good in the benchmark we wrote alongside it" bar.

The "FTS-without-embeddings vise" — a result, not a complaint

the synthesis-as-consolidation arc is the most useful one to walk through, because it converted what felt like an open question into a structural impossibility result. the original idea was simple: every dream cycle, cluster the Hebbian-linked memory entries, write one synthesized insight per cluster, retrieve the insight when a query semantically matches the cluster. lots of papers prescribe a version of this; the KAOS-native adaptation looked obvious.

the LLM variant failed the gates because the synthesizer wrote bridging prose but paraphrased the technical rule tokens. so an "insight" might say "the function exits early when the input is invalid" instead of preserving the exact tokens raise ValueError that the query was going to look for. KAOS uses FTS without embeddings (hard-constraint), so an exact-substring miss is a permanent miss.

the extractive variant preserved every technical token verbatim, by construction. it failed the gates for the opposite reason: the spine contained raise ValueError but no semantic bridge to the query "what happens when input is bad". without embeddings the query never reaches the spine in the first place.

that is more valuable than another mechanism. it closes a class of proposal cleanly, which means we stop spending cycles on variants of it.

The other three PRs

three pieces of release-shaped work landed alongside the eval primitive.

PR-1 — proposer streaming closes a real P0

the claude_code provider had been making a blocking subprocess.run(timeout=300) call. in practice this meant: every meta-harness iteration burned the full timeout when the CLI took its time, and the search loop died on the timeout rather than continuing. that broke the meta-harness as a useful tool against the CLI proposer. v0.9 switches to asyncio.create_subprocess_exec with incremental proc.stdout.read(8192) gated by asyncio.wait_for(read_to) where read_to = min(idle_timeout, remaining_wall). an idle stall raises ProposerStalled (recoverable, the loop continues to the next iteration); the absolute wall raises TimeoutError (hard). new CLI for cheap regression catching:

$ kaos doctor proposer
Proposer smoke -- 1 provider(s) (wall=30.0s, idle=10.0s)
  ok            claude-sonnet            29583.7 ms   OK

PR-2 — the eval harness itself

covered above. one design call worth flagging: the module exports a Probe ABC, but the synthesis-consolidation bench from earlier in the cycle was not retrofitted to use it. demo_synthesis_consolidation_bench/ stays as it was — it is the audit trail for that REJECT and rewriting it would invalidate the trail. the apparatus that proves the harness is reusable is the new probe in PR-4. that is the right test of an abstraction: it survives second use, not the use it was authored against.

PR-3 — experiments journal closes the queryability gap

when a verdict lands, nothing in KAOS used to record it durably alongside the git sha, the lock hash, and the per-arm stats. the next person had to grep commits. v0.9 adds one additive table (experiments, schema v8 → v9, no destructive changes) and four CLI commands:

kaos experiment log     --name --family --verdict --lock-sha256 \
                        --arms-json --gates-json --results-path
kaos experiment list    [--name] [--family] [--verdict-prefix] [--limit]
kaos experiment show    EXP_ID
kaos experiment compare A_ID B_ID

compare returns a changed-fields map so "what's new since the last run" becomes a one-line question. git_sha auto-fills from git rev-parse HEAD at log time.

PR-4 — the probe that earned its VOID

the strongest of the rejected candidates was Life-Harness's pre-execution Action Realization Layer — deterministic validation and canonicalization of malformed tool calls before the tool boundary. the one place KAOS v0.8.3 cannot trivially absorb is timing: the localizer fires after failure; Action Realization would fire before. that is a real net-new slice and the only one worth a probe.

the gates were locked at sha256 3ca89983... in commit fb6d579, before any probe code existed. they require +4.0pp over the v0.8.3 native baseline on the action-class slice with bootstrap95 lower bound above zero, +3.0pp over the never-fire lesion and +2.0pp over the random-fire lesion for causal isolation, no more than −1.0pp regression on the non-action control slice, and p95 inline overhead under 500 µs. workload is organic only — the lock file explicitly forbids synthetic substitution.

the falsification self-test passed (FULL := B1 emits [KILL: G1]; the harness can in fact kill the feature). the binding run on the local KAOS database returned:

[lock] ISA.lock.json sha256=3ca89983...
[workload] action=2  control=0  sanity=500
[VERDICT] VOID#1: insufficient organic action-class sample:
          n_action=2 < 200. Lock forbids synthetic substitution;
          collect more organic data.

that is the actual on-disk verdict. it is recorded in demo_action_realization_bench/VERDICT.md alongside the audit trail. the mechanism is not REJECTED — it is un-evaluated under the binding probe. it stays parked. when a live KAOS deployment accumulates 200 organic action-class incidents, the same probe can be re-run against the same lock hash and produce a binding feature verdict. v0.10 is not pre-committed to the mechanism.

What it looks like in code

the canonical example is examples/falsifiable_probe.py. it builds a tiny "does adding 4 few-shot examples help classification accuracy" probe end-to-end — pre-registered manifest, hash-lock, falsification self-test, binding run, journal log:

from kaos.eval.harness import Probe, ArmResults, GateOutcome, \
    bootstrap_diff_ci, compute_verdict
from kaos.experiments import ExperimentStore

class FewshotProbe(Probe):
    lock_path = "ISA.lock.json"
    known_sha256 = {"5aa9c10d...": "v1-demo"}

    def arms(self):
        return ["B0", "L1", "FULL"]

    def gates(self, arms):
        a_full = arms["FULL"].labels({"hard"})
        a_b0   = arms["B0"].labels({"hard"})
        md, lo, hi = bootstrap_diff_ci(a_full, a_b0, iters=500)
        g1 = md >= 0.10 and lo > 0.0
        diff_l1 = arms["FULL"].acc({"hard"}) - arms["L1"].acc({"hard"})
        g2 = diff_l1 >= 0.05
        return [
            GateOutcome("G1", "beats baseline", g1, kill=True,
                        detail=f"FULL-B0={md:+.3f}, lo={lo:+.3f}"),
            GateOutcome("G2", "causal isolation", g2, kill=True,
                        detail=f"FULL-L1={diff_l1:+.3f}"),
        ]

    def run(self, *, out_dir, **kw):
        arms = self._build_arms()
        outs = self.gates(arms)
        return {"verdict": compute_verdict(outs, judge_kappa=1.0),
                "judge_kappa": 1.0, "arms": {...}, "gates": [...]}

probe = FewshotProbe()
_, falsify_verdict = probe.falsify()    # must REJECT, else inadmissible
result = probe.run(out_dir="./out")     # ACCEPT / REJECT / VOID

with ExperimentStore("kaos.db") as store:
    store.log_run(name="fewshot-helps", verdict=result["verdict"],
                  lock_sha256=lock_sha, arms=result["arms"],
                  gates=result["gates"])

the deliberate shape of the API is that the rule for ACCEPT/REJECT/VOID lives in compute_verdict, which the probe author cannot override. that is the one bit of opinionation in the module — no probe gets to invent a softer verdict rule mid-run.

What v0.9 is not

it is worth being precise about what this release does not claim. it does not claim that KAOS is now provably bug-free. it does not claim that the discipline catches all bad mechanisms — it catches the ones whose effects are too small or too narrow to clear the bar, which is most of them but not all. it does not claim that REJECT/VOID was a foregone conclusion for any of the six candidates — the gates were authored before the results were known and each one could have ACCEPTed.

what it does claim is narrower: every claim a future KAOS release makes about a new mechanism can now be backed by a probe that the author of the mechanism could not have rigged. that is a quiet thing to ship, and it is the only thing v0.9 ships on purpose.

Try it

uv sync
uv run python examples/falsifiable_probe.py

uv run kaos doctor proposer
uv run kaos eval probe falsify \
    --probe demo_action_realization_bench.probe_adapter:ActionRealizationProbe
uv run kaos experiment list --db kaos.db

the CHANGELOG entry for v0.9.0 is here. the roadmap doc that frozen-pinned all four PRs and the explicit "no" list before any of them landed is here — useful to read as a record of how the discipline was applied to its own scope.

v0.10 is unscoped. it will not be scoped until at least one mechanism candidate produces an ACCEPT under this primitive.

v0.9.1 epilogue — MCP exposure

v0.9.0 deliberately did not expose the new CLI groups through MCP. that call held for about two days. it took roughly one real conversation with an agent — "what mechanisms have we already evaluated against the v0.8.3 baseline?" — to realize the read-side of the experiment journal was a legitimate mid-conversation need, not a release-cut affordance. shelling out via Bash works but is a paper cut every time.

v0.9.1 wraps all eight v0.9 surfaces as MCP tools. surface goes from 50 to 58:

each tool returns errors as JSON strings rather than crashing the dispatcher — the same containment rule the existing 50 tools follow. the rigor isn't only at the gate-design layer; the MCP wrappers carry 20 new tests in tests/test_mcp_v091.py that exercise the FULL dispatch path (list_tools + call_tool, JSON serialization, error containment) against the action-realization probe adapter and a real ExperimentStore. catches schema mismatches and wiring bugs that unit-testing the underlying primitives misses.

one small wiring fix worth flagging: _dispatch had a blanket assert _ccr is not None at the top. the v0.9.1 tools don't need _ccr (only _afs), so the assertion is now per-branch on the tools that actually orchestrate agents. it was always wrong to have it at the top; v0.9.1's tests just made that obvious.

— danilo

Papers cited: SWE-Skills-Bench · SAGE · Contextual Agentic Memory · AutoResearchClaw · HASP · Life-Harness