Auditing Discovery Claims: A Two-Sided Criterion for Agentic Science, with the Negative Side Decidable
On the map: LLM agents and tool use → Web, search, and research agents
Abstract (arXiv)
When a self-improving AI-for-science system claims a new capability, the evidence is usually a benchmark delta, a description-length gate, or a p-value. None separates a real gain from extra search, from a changed verifier, or from adaptation to a fallible oracle. We build a two-sided audit whose negative side is a formal fact: a pseudoknot-free oracle provably cannot represent a crossing base pair, so the prior verifier's range is bounded exactly, offline, before any run. "New" is relative to the agent's prior self, never to the base model. First, how far a single fallible oracle can inflate a capability claim. An invented, solver-free operator solves 43/60 crossing RNA targets under the predictor it optimizes, above a context-free floor of 0/60; under three predictors, 1/60 survives. Paired on the same 43 targets, a predictor the operator never saw confirms 2 of its designs against 26 for a minimum-free-energy solver (p = 8e-7). No statistic computed from the system and its own oracle sees that gap. Second, agent-written procedures can beat a human-written one under a judge no objective can flatter, at a fraction of the compute. Of six frontier models, the two whose operators ran without timeouts carry over at 0.293 against our 0.095 (n = 951 paired units, target-clustered [+0.108, +0.297], p = 5e-5) while spending 4.6-10x fewer oracle calls. Three rungs: difference under an outside adjudicator (reached), not bought with compute (reached, both directions), mechanism identified and transferable (not reached; seven candidates tested, none moves the statistic). The ceiling is the panel itself: its three predictors share nearest-neighbour thermodynamic parameters, two agreeing at kappa = 0.673. The audit is as unsparing about our own system: matched undirected search is an exact zero, and a search-free probe puts 84% of our headline effect on targets a random sequence already solves.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. First posted on arXiv 2 Aug 2026.
Builds on
0 of 0 references.
No references here yet.
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026From Fluent to Verifiable: Claim-Level Auditability for Deep Research Agents
- 2026Toward Auditable AI Scientists: A Hypothesis Evolution Protocol for LLM Agents
- 2026CrossAudit: A Git-Native, Cross-Vendor Audit Loop for Agentic Science
- 2026Beyond Execution: Auditing Experimental Fidelity in LLM-Driven Scientific Research
- 2026VST: Verifiable Structured Transport for Auditable Agent-to-Agent Alpha Discovery
- 2026Triage-PiEvo: Evidence Admission for Principle-Evolving Scientific Agents
- 2026Who Audits Whom, on What Substrate, with What Evidence? An Independence-Graded Audit Protocol for Agentic AI
- 2026Cheap to Hypothesize, Costly to Verify: The Defense Surface of Agentic Vulnerability Discovery