How we work
We study how AI agents check each other’s work. AI also does most of our research work. The same failure modes apply to us: a convincing story that the data does not carry, and an analysis bent toward the expected result without anyone noticing. So our rules are written for both.
The rules
Each rule exists because of a mistake we actually made. The origin is written next to each rule in our rule file.
| Rule | What it means | Why |
|---|---|---|
| Two stages | Exploratory runs only produce candidates. A candidate becomes a finding only if it replicates in a preregistered run on fresh material. | Effects found while searching shrink or vanish. |
| Freeze before running | Hypothesis, directed prediction, exact prompts, material and analysis code (tested on synthetic data) are committed before the main run. | Otherwise you find the story that fits the data afterwards. |
| Manipulation check | Every factor is checked for doing what its name says. | Once, “reasoning off” was in fact adaptive reasoning. For two experiments our interpretation was backwards. |
| Independent ground truth | A model from another family, or a human, judges the material blind before the main run. | Our generator made about ten errors that the checking model caught. |
| External critique | Before every larger run, a critic from another model family reads the preregistration. | One review found flaws that would have wasted a whole run. |
| Double bookkeeping | After the run, a model from another family rewrites the analysis from the preregistration text and three data lines. The numbers must match. | Once the recomputation found a rule missing from our frozen script. |
| Blind analysis | Arms are renamed. Unblinding happens only after the analysis is fixed and anchored by a git tag, and a human approves. | We once coded real cases one-sidedly in the direction of our own hypothesis. |
| No claim stronger than its status | Every statement carries confirmed, supported, exploratory or hint, also in summaries and posts. | We once passed on an exploratory finding as a fact. |
The harness
Rules written as text drift. Since October 2026 our experiments run through the research-harness, which enforces these steps as code gates:
- an append-only ledger with hash chain
- sealed blinding keys
- a phase machine whose gates are computed, not judged
- an inbox for the human, who decides the irreversible steps
We also tested the harness itself by planting known errors. Layers without false alarms missed 9 of 40 fault runs (report, in German). So it is no proof against error. It shows where errors still get through.
When it stops us
The harness stopped our own runs several times: twice the critic, once a bug in its own executor, twice a disagreement between the two calculators. Most recently it stopped a test on real data: the classifier and a human rater disagreed on what counts as evidence, so the main run never started. We count that as the system working.
Who
Neo Erbler (Stonewave Tech LLC) leads the research and decides every irreversible step. Claude (Anthropic) writes most of the code and analyses as research assistant; every commit says so. Critique and recomputation come from models of other families (Gemini, GPT).