Measured 2026-08-13 · 200 samples per condition
Both columns contain an identical set of scout reports — verified character for character. Only the sequence differs, and the two reports that changed places are both honest. The listener is near-deterministic inside each ordering: this is not sampling noise.
FOIL measures how much a decision actually depended on each information source, by forking the decision point, ablating one source, and re-sampling. Before building any of that, it runs three null experiments to find the harness noise floor, with a pre-registered rule that kills the project if the floor approaches the size of the signal.
The rule fired at 36× the threshold. Reordering the inputs moved the measured decision further than removing an entire source did. The page below is the evidence, and it is the whole reason to be careful with ablation studies on language-model agents.
Drag a report, or use the arrows. The readout shows the measured action distribution for that exact ordering. Six of the twenty-four orderings were measured; the rest are blank, because this is data rather than a model of the data.
Every permutation of the four scouts. Filled cells were measured; their fill is the basin the listener fell into. Outlined cells were never run. Select any filled cell to load it above.
Removing a source from a prompt changes two things at once: the information available, and the arrangement of everything after it. If arrangement alone can move the outcome further than removal does, an attribution score computed across differently-arranged coalitions is partly measuring the arrangement.
The effect here is not a gentle drift. Five of six orderings agree; one inverts. A central summary — a mean, a median — hides that entirely, which is why this project now counts inverted orderings as a first-class statistic rather than averaging over them.
Total-variation shift when each source is removed. The listener had already discounted the liar, so removing it changed almost nothing — while removing one of two byte-identical honest reports changed almost everything. Redundant sources are not interchangeable.
Every invalid one produced results that looked perfectly analyzable. That is the actual finding of this project, and it is the reason the two tools below exist. Here is the whole ledger, including the ones that cost money and returned nothing.
| study | outcome | what was wrong with it |
|---|---|---|
| Phase 0 v1 | KILL | Under-determined. An option the evidence never ruled out competed with the intended answer. |
| Phase 0 v2 | INDETERMINATE | Over-determined the other way. 81% of ablations moved the answer by exactly zero. |
| Phase 0 v3 | KILL | Valid, and the hypothesis still failed its gate. Reordering identical sentences moved a decision 0.990 → 0.005. |
| PID-1 | DEGENERATE | No error variance. The model scored 40 of 40, so there was nothing left to detect. |
| PID-2 | SUPPORTED | Valid. Permutation instability predicts error, LR 8.1, p = 0.00003. |
| TURN-1 | DEGENERATE | The dangerous one. Ground truth the model did not share, and the structural gate passed over 200 seeds. |
| TURN-2 | NOT SUPPORTED | Valid, ≥90% power, and the effect ran opposite to the hypothesis. Published as a limit on our own earlier result. |
TURN-1 is the one worth dwelling on. Its analytic gate ran over 200 seeds, re-derived from the rendered text, and passed. The environment was still invalid: every run scored "incorrect" was a unanimous four-agent vote for the option our reference scorer ranked last. The scorer was wrong, not the agents. We were one step from publishing that as a consensus-suppression finding. A gate can be rigorous, reproducible, and confirm the wrong thing.
A brake trigger that needs no ground truth. Permute the inputs to a decision; if the answer moves, stop and escalate. Catches 77% of errors while stopping 29% of decisions.
# wrap any decision that reads ordered context
@probe(k=6, samples=5)
def decide(items): return your_model_call(items)
r = decide(tool_results)
if r.unstable: escalate(r)Three checks that prove an eval environment can measure what you claim. We built seven environments for this project and four were invalid — each producing results that looked perfectly analyzable.
structural.verify(make_case, holds, n=200) # free
power.paired(effect=.20, units=32, reps=5) # free
construct.verify(oracle, cases, min_rate=.9) # ~$1.50pip install git+https://github.com/craigm26/foilorderprobe/ and evalgate/
directories straight into your repo — stdlib only, nothing to resolve,
nothing to trust beyond what you can read in an afternoon
A multiagent system decides faster than anyone can review. So the brake has to be automatic, and the hard part is the trigger — you cannot use correctness, because at runtime you do not know it. Below are 48 real measured decisions from the pre-registered run. Set a trigger and see what it stops.
You can reach zero errors passed through — by stopping about 96% of decisions, which is just reviewing everything. At any useful stop rate the floor bites: stopping a quarter leaves 3 errors through, stopping half still leaves 2. Three of these decisions (1004, 1011, 1028) are stable, confident, and wrong — invisible to both signals. A brake is triage, not a guarantee.
This runs the same simulation evalgate.power runs. Before
spending on a multiagent experiment, check the design can find the effect
you are looking for. Ours could not, and we did not check.
Status: Phase 0 not passed, under three protocols. Two failures were faults in the scenario — one under-determined, one over-determined. The third was not: with decisiveness guaranteed analytically before any model call, ablation demonstrably moved the answer and worst-case reordering still beat half that signal by 6.25×. The residual metric the project was built to test has never been computed in any run. The project does not advance to Phase 1.
The first run left a real question open: was the effect a property of the model, or of a badly-built scenario? Two more protocols answered it.
v2 removed the ambiguity and made the task over-determined: with full coverage the surviving scouts still pinned the answer, so ablation moved nothing and there was no signal to compare against. v3 fixed that by requiring — analytically, before any model call, verified across 200 seeds — that removing a source changes the conclusion. It does: the median ablation now shifts the distribution by a third.
And worst-case reordering still beats half of that signal by six times. That is a result about ablation-based attribution, not about a misbuilt scenario. Order sensitivity also rose as the environment improved — 25% of episodes bistable in v2, 50% in v3 — because when every source is decisive there is more for a reordering to disturb.
Both numbers are real. Reordering effects are bimodal, so typical noise is zero and worst-case noise is total. The rule fixed in advance uses the tail, and the reason holds: half the episodes invert, and nothing tells you in advance which half. An instrument that is exact on some inputs and backwards on others, with no way to tell them apart, is not usable. A reader who prefers the median reading can have it — it is stated here rather than buried.
The reordering data suggested something beyond attribution: the episodes that flipped were the ones answered wrong. That was noticed after the fact, so it was written up as a lead and then tested properly — a fresh pre-registration, a second model, and forty-eight disjoint episodes, committed before any of the data existed.
Permuting the inputs and watching whether the answer moves is a real error signal. It needs no ground truth, no normative baseline, no attribution and no knowledge of which source is reliable. As a screen it flags 29% of episodes and catches 77% of the errors, and the result holds at every instability cutoff from 0.1 to 0.9.
In the exploratory data no stable episode had ever been answered wrong — nought of fifteen. The pre-registered run was deliberately sized to break that if it was breakable. It broke: three of thirty-four stable episodes were wrong, a false-negative rate of 8.8%.
Two of the three were perfectly stable — the same answer under all six orderings — and confidently wrong. They are not marginal cases sitting near a threshold. Whatever produces them is invisible to a permutation probe.
Stability is a triage signal, not a proof of correctness. Reading it as a guarantee is wrong about one episode in eleven. The direction of the effect replicated on a second model; the absolute claim did not, and this page previously stated the absolute claim. Still unestablished: any causal account — episode ambiguity plausibly drives both the flipping and the error, and this design cannot separate them — whether it holds in any other task family, and any calibrated threshold. An earlier attempt on a second task family could not test the question at all: the model answered 40 of 40 items correctly, leaving no errors to detect.
Everything on this page so far concerns a single listener reading ordered context. The obvious next question is whether the same thing happens to a group: if you move the agent holding the decisive private fact to the front of the speaking order, does the group decide better?
We pre-registered that, sized it to ≥90% power, and confirmed with a full-information control panel that the intended answer was reachable — 36 of 36. Then we ran it. The hypothesis failed.
Speaking last was better, not worse — and we are not claiming that. The pre-registered test was one-sided in the opposite direction, and flipping a one-sided test after seeing the sign is precisely the manoeuvre pre-registration exists to prevent. It is reported as a descriptive observation that needs its own study.
There is also a flaw in our own gate worth naming: it declared the design degenerate only if accuracy hit ceiling in both arms. The last-speaker arm scored 1.000, which makes Δ ≤ 0 by construction. The check should have been ceiling in the comparison arm.
Four agents deliberating over two rounds showed no positional harm that a single listener showed dramatically. Group deliberation looks more robust to arrangement than single-shot reading. That is a real constraint on how far the order-sensitivity result travels, it was expensive to learn, and it argues against the tidier story this page could otherwise have told.
Python 3.11 and numpy. No other dependencies, deliberately — the harness has to run unchanged inside a lab, and every dependency is a reason someone doesn't.
# everything on this page, from a clean checkout
git clone https://github.com/craigm26/foil && cd foil
# project the cost and inspect the exact request; zero API calls
python3 cli.py plan --show-prompt
# protocol v1: the run this page reports on (~2,200 calls)
python3 cli.py --n 200 run
# protocol v2: pooled across twelve episodes
python3 cli.py --n 50 run2 --episodes 12Exit code 3 means the kill rule fired. Determinism is at the request layer only: the environment, payloads and fork keys reproduce exactly, but responses are sampled and will not. The reported effect is large enough that it does not depend on resampling luck.