mlsanityData to 7 Oct 2026

Capable but Unreliable: Canonical Path Deviation as a Causal Mechanism of Agent Failure in Long-Horizon Tasks

arXiv 2602.19008 ↗First posted 22 Feb 2026cs.CL cs.LGcited by 5

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Why do language agents fail on tasks they are capable of solving? We argue that many such failures are reliability failures caused by stochastic drift from a task's latent solution structure, not capability failures. Every well-defined tool-use task imposes a canonical solution path (i.e., a convergent set of tool invocations shared across successful runs) and agent success depends critically on whether a trajectory stays within this path's operating envelope. We establish this causally using a natural experiment that holds model capability and task difficulty fixed by construction. We analyze trajectories from the Toolathlon benchmark: 22 frontier models each attempt 108 real-world tool-use tasks across 3 independent runs, yielding 515 model×task units where the same model succeeds on some runs and fails on others due to LLM sampling stochasticity alone. Within these units, successful runs adhere significantly more closely to the canonical solution path than failed runs (+0.060 Jaccard, p<0.0001, n=488 units, 95% CI [+0.043, +0.077]). This result survives six robustness checks including cross-model-family leave-one-out validation. Critically, the causal mechanism is gradual and self-reinforcing: the adherence gap is statistically indistinguishable from zero through the first 50% of the trajectory, ruling out early-branching selection bias, and each off-canonical tool call raises the probability that the next call is also off-canonical by 22.7 percentage points (hatβ=+0.227, p<0.0001), more than doubling the baseline rate. These findings imply that agent reliability cannot be improved by capability scaling alone, but offer a highly actionable intervention: a simple monitor that restarts the bottom tercile of runs based on mid-trajectory canonical adherence lifts success rates by +8.8 percentage points among intervened runs.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 9 of its 20 references that are in the map. First posted on arXiv 22 Feb 2026.

Builds on

9 of 20 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution29 Oct
  2. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun

Other references · 7

  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  3. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  4. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  5. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  6. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  7. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 308

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making
  2. 2026Measuring Cross-Task Behavioral Consistency in Language Model Agents
  3. 2026A Wrong Turn Does Not Ruin the Journey: Deviation-Guided Skill Self-Evolution for LLM Agents
  4. 2026Loud Failures, Quiet Failures: Fault Detection and Recovery in Tool-Using Language Model Agents
  5. 2026Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure
  6. 2026Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents
  7. 2026CurveShift: Is Agent Progress Scalar? Separating Level from Shape
  8. 2026The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break