We Asked Six Models a Question. Every One That Answered Agreed. The Bug Was in the Question.

We were designing a session-bootstrap gate for a coding agent, the kind of system that briefs an AI on where it is and what it can trust before it touches anything. Part of the design needed sharpening, so we did what we usually do: put the design in front of a panel of models from different labs and asked them to attack it.

Then we asked a second question in the same sitting: name mechanisms in three unrelated source domains, ancient texts far from software, that this design could borrow from. Six models, five distinct labs, one prompt. Three of the six returned a formatted answer; two ran out of budget while reasoning and one failed to route. All three that answered picked the same three mechanisms, out of an enormous space of possible answers, and as far as we could see at that moment they had done so independently.

That looked like a finding: several systems nobody coordinated landing in the same place. We reported it internally as unprompted convergence, and were minutes from recording it that way.

The five-minute check that should have been the first minute

Before recording anything, we grepped our own prompt for the three answers the panel gave back.

They were there. Verbatim, more or less, sitting in the "here are some examples" section of the question we'd asked. We hadn't posed an open question and watched several labs discover the same thing independently. We'd handed the readers a multiple-choice question with the correct answers printed above it, and watched them read.

This is not an exotic mistake. It's the plainest version of a bias every survey designer, every UX researcher, and every scientist running a blinded trial already knows by name: a leading question doesn't measure the respondent, it measures the question. What made it easy to miss here is the setting. When the "respondents" are AI systems from different companies, agreement between them carries a kind of borrowed authority that agreement between people reading the same brochure doesn't. It looks like corroboration. It is the same leading-question artifact.

Running the control

So we ran the same panel again, on the same design problem, with the examples removed. Same target design, same three source domains, named only by category, no candidate mechanisms, no suggested mappings, explicit instructions not to hedge toward the "obvious" answer.

Four usable outputs came back: two full answers, and two reasoning traces from seats that ran out of budget before formatting a reply but whose working could still be read for drift. None of the four overlapped with the original seeded trio. Between themselves they overlapped almost nowhere; one broad motif recurred loosely in two of them, and it is among the most famous patterns in that corpus, the kind any well-read system would reach for regardless of priming. One model, given the open framing without examples to lean on, declined to answer rather than manufacture a confident pick, which is its own useful data point, not a failure to log and forget.

That's the actual result. Not "the panel discovered something," but "the panel repeats whatever you show it, and stops the moment you stop showing it." This does not make every agreement leakage. Independently trained systems will sometimes converge on the most salient feature of a rich domain, and that loose recurrence looked nothing like the seeded run: sparse, phrased differently by each system, and explainable by how common the pattern is. The seeded agreement was dense, phrased in our own words, and traceable to a line in our prompt. That difference is the test.

What this means if you run panels for anything real

If you evaluate outputs across multiple models, or run any kind of AI jury to check a design, a claim, or a piece of code, this is worth checking every time, not just when the result looks suspicious.

Read your own prompt for your own answer. Before treating multi-model agreement as evidence, grep the shared context every model saw for the specific conclusion they all reached. If it is in there, you have measured your prompt's clarity, not the models' judgment.

Examples are not neutral. "For instance, X could work like A, B, or C" is the same instrument as a multiple-choice question, and every model that reads it will treat it that way, correctly, because that is what reading comprehension is for. If the panel's job is discovery rather than validation, name the domain and stop there. The moment you illustrate the target, you have handed over the target.

A blind control is cheap and non-negotiable for anything you are about to act on. We reran the identical question with the seeding stripped out. It took one more panel run. That is a trivial cost against building an architecture, or worse, a public claim, on a false convergence signal.

Refusal to answer is data, not noise. One model in the blind run said, in effect, that it did not have enough to commit to three specific mechanisms without guessing. We almost treated that as a non-response. It is closer to the correct behavior under genuine uncertainty than a confident pick manufactured to fill the slot would have been.

What actually held up

The underlying engineering ideas we were testing for did not depend on the seeded run: a hard reset boundary instead of an indefinitely extending trust chain, an internal tamper-evident sequence counter instead of trust in the host clock, and one-shot sealed adjudication with no silent retry-until-pass. They did not need the source material that seeded the contaminated run to be defensible; they stand on their own engineering merits, and we built the gate on them. What we threw out was the story that independent minds had discovered them at the same moment. Nobody discovered anything. We asked a leading question and were, briefly, convinced by our own leading question.

This post is not a discovery report. It is the record of catching the version of it that would have been wrong, and of the check we almost skipped.