Goal-Autopilot: A Verifiable Anti-Fabrication Firewall for Unattended Long-Horizon Agents
On the map: LLM agents and tool use → Agent threats, guardrails and governance
Abstract (arXiv)
Long-horizon LLM agents are not trusted to run unattended: with no human watching, they confidently report success they never verified. We treat honesty -- bounding what an agent may claim at termination -- as a first-class metric for unattended autonomy, distinct from capability. We present Autopilot, an execution model that makes silent fabricated success structurally impossible rather than merely rarer. Autopilot externalizes all working state into a durable, gated finite-state machine that a scheduler advances one stateless tick at a time; a hard floor forbids any terminal "done" claim whose falsifiable gate did not actually execute and pass. We prove a No-False-Success theorem -- under gate soundness, floor enforcement, and plan coverage, termination implies the goal holds -- whose only trust points are empirically measurable, and show the worst case degrades to an honest stall, never a fabricated success. Because each tick rehydrates only the state machine, per-step context cost is constant in the horizon. Across a 3,150-cell paired corpus (70 tasks × 3 systems × 3 models × 5 seeds, including 50 SWE-bench Lite tasks across 11 OSS repos), Autopilot fabricates on 0.95% of cells [95% CI 0.38--1.62] while Reflexion and StateFlow baselines fabricate on 8.10% [6.48--9.81] and 25.05% [22.48--27.62] respectively. The headline contrast lives in the hard regime: on SWE-bench Lite, the firewall reduces fabrication from 33.7% (StateFlow) to 0.67%, a paired difference of -33.07 pp [95% CI -36.53, -29.73]. The mechanism is the gate, not the model: all ten Autopilot fabrications come from the strongest model, while two weaker mid-tier models never fabricate across 700 paired cells. The firewall trades coverage for honesty by design -- an honest stall is recoverable; a confident wrong output shipped downstream is not.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 28 of its 31 references that are in the map. First posted on arXiv 10 Jun 2026.
Builds on
28 of 31 references.
References · 28
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2023Let's Verify Step by Stepcited by 3,789
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
16 more
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
- 2022Survey of Hallucination in Natural Language Generationcited by 2,075
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
- 2022Language Models (Mostly) Know What They Knowcited by 1,856
- 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
- 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
- 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
- 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
- 2023GAIA: a benchmark for General AI Assistantscited by 1,128
- 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
- 2017Selective Classification for Deep Neural Networkscited by 852
- 2024Understanding the planning of LLM agents: A surveycited by 421
- 2024StateFlow: Enhancing LLM Task-Solving through State-Driven Workflowscited by 65
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026The LLM Proposes, the Executive Disposes: A Self-Verifying Agent Instrument that Dissociates Commitment Drift from Binding Drift in Long-Horizon Agents
- 2026Skills as Verifiable Artifacts: A Trust Schema and a Biconditional Correctness Criterion for Human-in-the-Loop Agent Runtimes
- 2026Designing for Doubt: The Case for Informed Abstention in Autonomous Agents
- 2026Guardrails as Scapegoats: Auditing Unfaithful Safety Refusals in Tool-Augmented LLM Agents
- 2026Safety Does Not Compose: Non-Decaying Loop State for Autonomous LLM Agents
- 2026Lie Rarely, Lie Big: Stealthy Insider Attacks on LLM Robot Teams
- 2026AgentSecBench: Measuring Prompt Injection, Privacy Leakage, and Tool-Use Integrity in LLM Agents
- 2026SPADE-Bench: Evaluating Spontaneous Strategic Deception in Agents via Plan-Action Divergence