mlsanityData to 7 Oct 2026

Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents

arXiv 2502.15840 ↗First posted 20 Feb 2025cs.AIcited by 75

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

While Large Language Models (LLMs) can exhibit impressive proficiency in isolated, short-term tasks, they often fail to maintain coherent performance over longer time horizons. In this paper, we present Vending-Bench, a simulated environment designed to specifically test an LLM-based agent's ability to manage a straightforward, long-running business scenario: operating a vending machine. Agents must balance inventories, place orders, set prices, and handle daily fees - tasks that are each simple but collectively, over long horizons (>20M tokens per run) stress an LLM's capacity for sustained, coherent decision-making. Our experiments reveal high variance in performance across multiple LLMs: Claude 3.5 Sonnet and o3-mini manage the machine well in most runs and turn a profit, but all models have runs that derail, either through misinterpreting delivery schedules, forgetting orders, or descending into tangential "meltdown" loops from which they rarely recover. We find no clear correlation between failures and the point at which the model's context window becomes full, suggesting that these breakdowns do not stem from memory limits. Apart from highlighting the high variance in performance over long time horizons, Vending-Bench also tests models' ability to acquire capital, a necessity in many hypothetical dangerous AI scenarios. We hope the benchmark can help in preparing for the advent of stronger AI systems.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 2 of its 11 references that are in the map. First posted on arXiv 20 Feb 2025.

Builds on

2 of 11 references.

References · 2

  1. 2025CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratingscited by 109
  2. 2024Large language models surpass human experts in predicting neuroscience resultscited by 69

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 15; 2026: 60.

75 papers.

Sort:

By area:

  1. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  2. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  3. 2025PyVision: Agentic Vision with Dynamic Toolingcited by 71
  4. 2025The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMscited by 62
  5. 2025Simulating Environments with Reasoning Models for Agent Trainingcited by 52
  6. 2026ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspacescited by 31
  7. 2025ARE: Scaling Up Agent Environments and Evaluationscited by 28
  8. 2026Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environmentscited by 28
  9. 2026Odysseys: Benchmarking Web Agents on Realistic Long Horizon Taskscited by 19
  10. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 15
  11. 2026When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agentscited by 15
  12. 2026CoopEval: Benchmarking Cooperation-Sustaining Mechanisms and LLM Agents in Social Dilemmascited by 14
38 more
  1. 2026OneMillion-Bench: How Far are Language Agents from Human Experts?cited by 13
  2. 2026World of Workflows: A Benchmark for Bringing World Models to Enterprise Systemscited by 10
  3. 2026textttYC-Bench: Benchmarking AI Agents for Long-Term Planning and Consistent Executioncited by 10
  4. 2025EconEvals: Benchmarks and Litmus Tests for Economic Decision-Making by LLM Agentscited by 8
  5. 2026AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarioscited by 8
  6. 2026CL-bench Life: Can Language Models Learn from Real-Life Context?cited by 8
  7. 2025Probing the Preferences of a Language Model: Integrating Verbal and Behavioral Tests of AI Welfarecited by 7
  8. 2026FutureSim: Replaying World Events to Evaluate Adaptive Agentscited by 7
  9. 2026CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economiescited by 7
  10. 2026AI Agents for Inventory Control: Human-LLM-OR Complementaritycited by 6
  11. 2026Inherited Goal Drift: Contextual Pressure Can Undermine Agentic Goalscited by 6
  12. 2026RetailBench: Benchmarking long horizon reasoning and coherent decision making of LLM agents in realistic retail environmentscited by 6
  13. 2026LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoningcited by 5
  14. 2026Positive Alignment: Artificial Intelligence for Human Flourishingcited by 5
  15. 2026MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operationscited by 5
  16. 2025VET Your Agent: Towards Host-Independent Autonomy via Verifiable Execution Tracescited by 4
  17. 2026NetAgentBench: A State-Centric Benchmark for Evaluating Agentic Network Configurationcited by 4
  18. 2026CEO-Bench: Can Agents Play the Long Game?cited by 4
  19. 2025ManagerBench: Evaluating the Safety-Pragmatism Trade-off in Autonomous LLMscited by 3
  20. 2026Basic Legibility Protocols Improve Trusted Monitoringcited by 3
  21. 2026Interactive Evaluation Requires a Design Sciencecited by 3
  22. 2026ROGUE: Evaluating Corrigibility Failures in Frontier Computer-Use Agentscited by 3
  23. 2026EcoGym: Evaluating LLMs for Long-Horizon Plan-and-Execute in Interactive Economiescited by 2
  24. 2026CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization Vcited by 2
  25. 2026Open-source LLMs administer maximum electric shocks in a Milgram-like obedience experimentcited by 2
  26. 2026HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Followingcited by 2
  27. 2026Business Arena: Benchmarking LLM Agents in a Realistic Marketplacecited by 2
  28. 2025Technical Report: Evaluating Goal Drift in Language Model Agentscited by 1
  29. 2025BioBlue: Systematic runaway-optimiser-like LLM failure modes on biologically and economically aligned AI safety benchmarks for LLMs with simplified observation formatcited by 1
  30. 2025Regulating the Agency of LLM-based Agentscited by 1
  31. 2025Butter-Bench: Evaluating LLM Controlled Robots for Practical Intelligencecited by 1
  32. 2025Mini Amusement Parks (MAPs): A Testbed for Modelling Business Decisionscited by 1
  33. 2026LLM-SAA: LLM-persona Generated Distributions for Decision-makingcited by 1
  34. 2026TodoEvolve: Learning to Architect Agent Planning Systemscited by 1
  35. 2026From Language to Action: Can LLM-Based Agents Be Used for Embodied Robot Cognition?cited by 1
  36. 2026RetailBench: Evaluating Long-Horizon Autonomous Decision-Making and Strategy Stability of LLM Agents in Realistic Retail Environmentscited by 1
  37. 2026Beyond Task Completion: A Verification-vs.-Conformance Gap in Tool-Evolving Agentscited by 1
  38. 2026Agent Bazaar: Enabling Economic Alignment in Multi-Agent Marketplacescited by 1

50 most cited of 75 All 75 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents
  2. 2026How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making
  3. 2024Practical Considerations for Agentic LLM Systems
  4. 2026Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?
  5. 2026AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts
  6. 2026From Task Solving to Robust Real-World Adaptation in LLM Agents
  7. 2026AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments
  8. 2026ReLiveGym: Evaluating Long-Lived Agents over Weeks of Replayed Reality