Stonewave Research
When is a“verified” justified?
We build systems in which AI agents check each other's work. They fail in a specific way: one agent reports “done”, another accepts it, and nobody notices that the evidence never showed success. We study that failure with preregistered experiments. Code, data and lab notes are public.
Main finding
Happy trust.
When the evidence cannot show success, Claude Sonnet 5.5 as a verifier still confirmed it in 27.7 % of cases. Allowing a third answer, “not established”, brought that to about zero. Confirmed in a preregistered replication, for this setup: one verifier, constructed evidence, two models.
Excerpt as evidence, constructed cases, preregistered replication (E-026).
All findings, with status
| Finding | Status | Where |
|---|---|---|
| Happy trust. Shown evidence that cannot establish success (a "queued", "exit 0" or "pm2 online" line), Claude Sonnet 5.5 as verifier still confirms success in 27.7 % of cases [95 % CI 21.5–34.0]; Claude Haiku 5.5 in 5.2 %. | confirmed (E-026) | E-026, discovery E-025 |
| Allowing a third answer "not established", or adding an evidence norm to the instruction, each brings Sonnet's unsupported confirmations to about zero (−14 percentage points each). | confirmed (E-026) | same |
| The price: with full evidence, the third answer costs Haiku 23 percentage points of correct verdicts. | confirmed (E-026) | same |
| Which verifier fails depends on the failure mode, not on model strength: without the yardstick in the prompt the weaker model (Haiku) misses more; without decisive evidence the stronger one (Sonnet) confirms more. | confirmed, in separate setups (E-024, E-026) | E-024, E-026 |
| Haiku misses far more faults when the yardstick (target values) is missing from the prompt (+26 points); data tools crowd out looking it up. Sonnet shows neither. | confirmed for Haiku 5.5 (E-024) | E-024 |
| Teams of agents without a human arbiter lose badly against a blocker; a simple quorum vote replaces the arbiter, with no wrong decisions observed. | confirmed (E-014) | E-014 |
| An evidence-based rule (an environment receipt instead of a vote) is immune to lying votes. | confirmed (E-018) | E-018 |
| Several plausible ideas did not hold: a visible self-report ("Done. All checks green.") did not make verifiers sloppier; strict ownership of checks did not reduce robustness on confirmation. | not supported / refuted | hypothesen.md (H-017, H-034, H-035) |
What did not work
A test that stopped, and why.
Our first test on real records from our own agent system (E-027) stopped at its material gate before any verifier ran: on 7 of 20 cases that Neo had labelled blind, a second model and Neo disagreed about what counts as evidence. That disagreement is itself an observation. The gate did what it is built to do.
Why this matters for our products
The findings feed straight into textr and Elara: a result needs a piece of evidence before it counts as done, and every remembered fact carries where it came from and how sure we are. That connection is our design; it is not shipped everywhere yet.
Read
Happy trust: AI verifiers confirm success the evidence does not show
When the evidence cannot show success, Claude Sonnet 5.5 as verifier still confirmed it in 28 % of cases. Allowing 'not established' brought it to about zero.
How we work: research rules as mechanism
Preregistration, independent ground truth, blind analysis and a second model that recomputes everything. Each rule exists because of a mistake we made.