mlsanityData to 14 Oct 2026

Goal-Autopilot: A Verifiable Anti-Fabrication Firewall for Unattended Long-Horizon Agents

arXiv 2606.11688 ↗First posted 10 Jun 2026cs.CL cs.AIcited by 1

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Long-horizon LLM agents are not trusted to run unattended: with no human watching, they confidently report success they never verified. We treat honesty -- bounding what an agent may claim at termination -- as a first-class metric for unattended autonomy, distinct from capability. We present Autopilot, an execution model that makes silent fabricated success structurally impossible rather than merely rarer. Autopilot externalizes all working state into a durable, gated finite-state machine that a scheduler advances one stateless tick at a time; a hard floor forbids any terminal "done" claim whose falsifiable gate did not actually execute and pass. We prove a No-False-Success theorem -- under gate soundness, floor enforcement, and plan coverage, termination implies the goal holds -- whose only trust points are empirically measurable, and show the worst case degrades to an honest stall, never a fabricated success. Because each tick rehydrates only the state machine, per-step context cost is constant in the horizon. Across a 3,150-cell paired corpus (70 tasks × 3 systems × 3 models × 5 seeds, including 50 SWE-bench Lite tasks across 11 OSS repos), Autopilot fabricates on 0.95% of cells [95% CI 0.38--1.62] while Reflexion and StateFlow baselines fabricate on 8.10% [6.48--9.81] and 25.05% [22.48--27.62] respectively. The headline contrast lives in the hard regime: on SWE-bench Lite, the firewall reduces fabrication from 33.7% (StateFlow) to 0.67%, a paired difference of -33.07 pp [95% CI -36.53, -29.73]. The mechanism is the gate, not the model: all ten Autopilot fabrications come from the strongest model, while two weaker mid-tier models never fabricate across 700 paired cells. The firewall trades coverage for honesty by design -- an honest stall is recoverable; a confident wrong output shipped downstream is not.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 28 of its 31 references that are in the map. First posted on arXiv 10 Jun 2026.

Builds on

28 of 31 references.

References · 28

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  9. 2023Let's Verify Step by Stepcited by 3,789
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  11. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  12. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
16 more
  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  3. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  4. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  5. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  6. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  7. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  8. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  9. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  10. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  11. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  12. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  13. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  14. 2017Selective Classification for Deep Neural Networkscited by 852
  15. 2024Understanding the planning of LLM agents: A surveycited by 421
  16. 2024StateFlow: Enhancing LLM Task-Solving through State-Driven Workflowscited by 65

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026The LLM Proposes, the Executive Disposes: A Self-Verifying Agent Instrument that Dissociates Commitment Drift from Binding Drift in Long-Horizon Agents
  2. 2026Skills as Verifiable Artifacts: A Trust Schema and a Biconditional Correctness Criterion for Human-in-the-Loop Agent Runtimes
  3. 2026Designing for Doubt: The Case for Informed Abstention in Autonomous Agents
  4. 2026Guardrails as Scapegoats: Auditing Unfaithful Safety Refusals in Tool-Augmented LLM Agents
  5. 2026Safety Does Not Compose: Non-Decaying Loop State for Autonomous LLM Agents
  6. 2026Lie Rarely, Lie Big: Stealthy Insider Attacks on LLM Robot Teams
  7. 2026AgentSecBench: Measuring Prompt Injection, Privacy Leakage, and Tool-Use Integrity in LLM Agents
  8. 2026SPADE-Bench: Evaluating Spontaneous Strategic Deception in Agents via Plan-Action Divergence