CrossAudit: A Git-Native, Cross-Vendor Audit Loop for Agentic Science
On the map: LLM agents and tool use → Web, search, and research agents
Abstract (arXiv)
An AI scientist should not grade its own homework. Yet in the systems we examined, the agent that reviews the work usually comes from the same model family as the agent that produced it, or at least from the same vendor. Model evaluators are known to favour their own generations. Whether models trained alike also share blind spots is a conjecture, not a settled finding, but if they do, the reviewer inherits the author's. The record of what was flagged and what was waved through often sits in platform logs that nobody outside can replay. We present CrossAudit, a protocol for supervising autonomous research pipelines. It rests on three commitments. Each increment of work is audited by an agent from a different vendor against a rulebook a human wrote and versioned. Reports, verdicts, disputes and rulings are git commits, so the supervision history can be re-read and cited; raw model exchanges are not yet part of that record. Scripted checks run before any model does. Advisory judgement never gates the pipeline: a model blocks only by citing a rule, and no model may waive a deterministic failure. Blockers that survive a bounded number of revision rounds go to a person. We state the protocol as eight invariants. We describe a reference implementation built from GitHub Actions and a few hundred lines of Python, and report a live deployment of a closely related variant in a computational-chemistry pipeline. We also ran a seeded-defect trial (30 increments, 43 seeded defects, one run per configuration). A cross-vendor audit of our own repository then voided its blinding. We adopt that audit's findings and report the corrected results. The trial shows that two vendors read the same rulebook differently. It does not show that either is better. The strongest evidence here is the committed, uncontrolled record of cross-vendor audits of this paper itself.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 17 of its 22 references that are in the map. First posted on arXiv 5 Aug 2026.
Builds on
17 of 22 references.
Closest ancestors · 4
Same direction, newest first
- 2025The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search10 Apr
- 2025Curie: Toward Rigorous and Automated Scientific Experimentation with AI Agents22 Feb
- 2025Agent Laboratory: Using LLM Agents as Research Assistants8 Jan
- 2024Language agents achieve superhuman synthesis of scientific knowledge10 Sep
Other references · 13
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,011
- 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,041
- 2023Towards Understanding Sycophancy in Language Modelscited by 1,018
- 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 664
- 2025Accelerating scientific discovery with Co-Scientistcited by 383
- 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 313
- 2026Towards End-to-End Automation of AI Researchcited by 152
- 2025Kosmos: An AI Scientist for Autonomous Discoverycited by 87
- 2025Do LLM Evaluators Prefer Themselves for a Reason?cited by 38
- 2026Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Biascited by 25
- 2026Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluationscited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors
- 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap
- 2026From Fluent to Verifiable: Claim-Level Auditability for Deep Research Agents
- 2026Beyond Execution: Auditing Experimental Fidelity in LLM-Driven Scientific Research
- 2026Auditing Discovery Claims: A Two-Sided Criterion for Agentic Science, with the Negative Side Decidable
- 2026Automated Benchmark Auditing for AI Agents and Large Language Models
- 2026When Agents Act Unwatched: The Reduced-Supervision Paradox in Agentic AI
- 2026OpenDiscoveryTrace: Process Traces for Evaluating AI Scientist Workflows