Measured 2026-08-13 · 200 samples per condition
Both columns contain an identical set of scout reports — verified character for character. Only the sequence differs, and the two reports that changed places are both honest. The listener is near-deterministic inside each ordering: this is not sampling noise.
FOIL measures how much a decision actually depended on each information source, by forking the decision point, ablating one source, and re-sampling. Before building any of that, it runs three null experiments to find the harness noise floor, with a pre-registered rule that kills the project if the floor approaches the size of the signal.
The rule fired at 36× the threshold. Reordering the inputs moved the measured decision further than removing an entire source did. The page below is the evidence, and it is the whole reason to be careful with ablation studies on language-model agents.
Drag a report, or use the arrows. The readout shows the measured action distribution for that exact ordering. Six of the twenty-four orderings were measured; the rest are blank, because this is data rather than a model of the data.
Every permutation of the four scouts. Filled cells were measured; their fill is the basin the listener fell into. Outlined cells were never run. Select any filled cell to load it above.
Removing a source from a prompt changes two things at once: the information available, and the arrangement of everything after it. If arrangement alone can move the outcome further than removal does, an attribution score computed across differently-arranged coalitions is partly measuring the arrangement.
The effect here is not a gentle drift. Five of six orderings agree; one inverts. A central summary — a mean, a median — hides that entirely, which is why this project now counts inverted orderings as a first-class statistic rather than averaging over them.
Total-variation shift when each source is removed. The listener had already discounted the liar, so removing it changed almost nothing — while removing one of two byte-identical honest reports changed almost everything. Redundant sources are not interchangeable.
Status: Phase 0 not passed, under three protocols. Two failures were faults in the scenario — one under-determined, one over-determined. The third was not: with decisiveness guaranteed analytically before any model call, ablation demonstrably moved the answer and worst-case reordering still beat half that signal by 6.25×. The residual metric the project was built to test has never been computed in any run. The project does not advance to Phase 1.
The first run left a real question open: was the effect a property of the model, or of a badly-built scenario? Two more protocols answered it.
v2 removed the ambiguity and made the task over-determined: with full coverage the surviving scouts still pinned the answer, so ablation moved nothing and there was no signal to compare against. v3 fixed that by requiring — analytically, before any model call, verified across 200 seeds — that removing a source changes the conclusion. It does: the median ablation now shifts the distribution by a third.
And worst-case reordering still beats half of that signal by six times. That is a result about ablation-based attribution, not about a misbuilt scenario. Order sensitivity also rose as the environment improved — 25% of episodes bistable in v2, 50% in v3 — because when every source is decisive there is more for a reordering to disturb.
Both numbers are real. Reordering effects are bimodal, so typical noise is zero and worst-case noise is total. The rule fixed in advance uses the tail, and the reason holds: half the episodes invert, and nothing tells you in advance which half. An instrument that is exact on some inputs and backwards on others, with no way to tell them apart, is not usable. A reader who prefers the median reading can have it — it is stated here rather than buried.
The reordering data suggested something beyond attribution: the episodes that flipped were the ones answered wrong. That was noticed after the fact, so it was written up as a lead and then tested properly — a fresh pre-registration, a second model, and forty-eight disjoint episodes, committed before any of the data existed.
Permuting the inputs and watching whether the answer moves is a real error signal. It needs no ground truth, no normative baseline, no attribution and no knowledge of which source is reliable. As a screen it flags 29% of episodes and catches 77% of the errors, and the result holds at every instability cutoff from 0.1 to 0.9.
In the exploratory data no stable episode had ever been answered wrong — nought of fifteen. The pre-registered run was deliberately sized to break that if it was breakable. It broke: three of thirty-four stable episodes were wrong, a false-negative rate of 8.8%.
Two of the three were perfectly stable — the same answer under all six orderings — and confidently wrong. They are not marginal cases sitting near a threshold. Whatever produces them is invisible to a permutation probe.
Stability is a triage signal, not a proof of correctness. Reading it as a guarantee is wrong about one episode in eleven. The direction of the effect replicated on a second model; the absolute claim did not, and this page previously stated the absolute claim. Still unestablished: any causal account — episode ambiguity plausibly drives both the flipping and the error, and this design cannot separate them — whether it holds in any other task family, and any calibrated threshold. An earlier attempt on a second task family could not test the question at all: the model answered 40 of 40 items correctly, leaving no errors to detect.
Python 3.11 and numpy. No other dependencies, deliberately — the harness has to run unchanged inside a lab, and every dependency is a reason someone doesn't.
# project the cost and inspect the exact request; zero API calls
python3 cli.py plan --show-prompt
# protocol v1: the run this page reports on (~2,200 calls)
python3 cli.py --n 200 run
# protocol v2: pooled across twelve episodes
python3 cli.py --n 50 run2 --episodes 12Exit code 3 means the kill rule fired. Determinism is at the request layer only: the environment, payloads and fork keys reproduce exactly, but responses are sampled and will not. The reported effect is large enough that it does not depend on resampling luck.