Hearsay: Can an Auditor Trust the Record a Deployed Agent Harness Writes?
Not on the map. Why?
Abstract (arXiv)
An agent harness, the code that turns a model into an agent, writes its own record of each run, and that record is all a later reader gets when a run is disputed, investigated or audited. We call a record evidentiary when a reader who was not there can check it without trusting the writer. Across sixteen deployed frameworks, none writes one in full. Hearsay examines the record, not the task: five harnesses run fourteen tasks, three blinded LLM examiners and a human panel read the records, and every excerpt an examiner quotes is checked mechanically for who wrote it. First, the record lets a reader name the fault but not prove how the run went. Examiners name the right fault in 74 to 91% of 140 runs, but the fault can be proved only from two files the benchmark adds; for what happened in between, fewer than one citation in ten lands on anything the harness did not write, and the examiner with the fewest false alarms catches half of the entries we delete, rewrite or fabricate. Second, the remedy is a second author, not a stronger seal on the first. An append-only log of what passes between harness and model, kept outside the harness and read against the record in both directions, reports all 28 omissions and fabrications we made a harness commit as it ran, where a hash chain over the harness's own record passes all 28. Handed the log, examiners keep their fault verdicts but rest more of their citations on what the harness did not write. What makes a record evidence is who writes it, not what is captured.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 41 references that are in the map. First posted on arXiv 26 Sep 2026.
Builds on
22 of 41 references.
References · 22
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
- 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
- 2025Why Do Multi-Agent LLM Systems Fail?cited by 604
- 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 238
- 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systemscited by 185
- 2026Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnessescited by 142
- 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 126
- 2025AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?cited by 116
- 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 104
- 2024Visibility into AI Agentscited by 79
10 more
- 2026Dive into Claude Code: The Design Space of Today's and Future AI Agent Systemscited by 38
- 2026Seeing the Whole Elephant: A Benchmark for Failure Attribution in LLM-based Multi-Agent Systemscited by 27
- 2025AgentSight: System-Level Observability for AI Agents Using eBPFcited by 21
- 2017Practical Whole-System Provenance Capturecited by 19
- 2026Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architecturescited by 18
- 2026Agent-Diff: Benchmarking LLM Agents on Enterprise API Tasks via Code Execution with State-Diff-Based Evaluationcited by 13
- 2026Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agentscited by 10
- 2026AgentTrace: Causal Graph Tracing for Root Cause Analysis in Deployed Multi-Agent Systemscited by 7
- 2026Architectural Design Decisions in AI Agent Harnessescited by 4
- 2026DEMM-Bench: A Cross-Regime Benchmark for Agent-Runtime Governance-Evidence Sufficiencycited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Auditing Agent Harness Safety
- 2026What Does a Harness Buy? Tokens, Mostly
- 2026Clean Scores, Buried Evidence, and Confident Wrong: A Receipt-Based Audit of Frontier Agentic QA
- 2026CatchBench: When Can an Agent Failure Be Caught?
- 2026Quantifying Overclaiming Propensity in Frontier LLM Agents
- 2026From Runtime Records to Legal Findings: An Evidentiary-Adequacy Criterion for Agentic AI Oversight
- 2026Faithful Where It Can Be Checked: Auditing a Reflection Agent Against Its System Prompt in a Randomized Trial
- 2026LEDGER: Claim-to-Evidence Trace Graphs for Auditing LLM Agents