mlsanityData to 14 Oct 2026

Robotouille: An Asynchronous Planning Benchmark for LLM Agents

arXiv 2502.05227 ↗First posted 6 Feb 2025cs.RO cs.AI cs.CLcited by 32

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Effective asynchronous planning, or the ability to efficiently reason and plan over states and actions that must happen in parallel or sequentially, is essential for agents that must account for time delays, reason over diverse long-horizon tasks, and collaborate with other agents. While large language model (LLM) agents show promise in high-level task planning, current benchmarks focus primarily on short-horizon tasks and do not evaluate such asynchronous planning capabilities. We introduce Robotouille, a challenging benchmark environment designed to test LLM agents' ability to handle long-horizon asynchronous scenarios. Our synchronous and asynchronous datasets capture increasingly complex planning challenges that go beyond existing benchmarks, requiring agents to manage overlapping tasks and interruptions. Our results show that ReAct (gpt4-o) achieves 47% on synchronous tasks but only 11% on asynchronous tasks, highlighting significant room for improvement. We further analyze failure modes, demonstrating the need for LLM agents to better incorporate long-horizon feedback and self-audit their reasoning during task execution. Code is available at https://github.com/portal-cornell/robotouille.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 43 of its 45 references that are in the map. First posted on arXiv 6 Feb 2025.

Builds on

43 of 45 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents13 Aug
  2. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun
  3. 2024Can Graph Learning Improve Planning in LLM-based Agents?29 May
  4. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents24 Jan
  5. 2023Evaluating Language-Model Agents on Realistic Autonomous Tasks18 Dec
  6. 2023FireAct: Toward Language Agent Fine-tuning9 Oct
  7. 2023AgentBench: Evaluating LLMs as Agents7 Aug

Other references · 36

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  8. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  9. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  10. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  11. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  12. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
24 more
  1. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  2. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  3. 2024Executable Code Actions Elicit Better LLM Agentscited by 564
  4. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  5. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501
  6. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 462
  7. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  8. 2018VirtualHome: Simulating Household Activities via Programscited by 415
  9. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 382
  10. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 366
  11. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 349
  12. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 284
  13. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planningcited by 207
  14. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 200
  15. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 151
  16. 2023Can Large Language Models Really Improve by Self-critiquing Their Own Plans?cited by 108
  17. 2023T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Stepcited by 93
  18. 2024Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrappingcited by 90
  19. 2023MindAgent: Emergent Gaming Interactioncited by 88
  20. 2023Coupling Large Language Models with Logic Programming for Robust and General Reasoning from Textcited by 59
  21. 2023MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaborationcited by 57
  22. 2022Plansformer: Generating Symbolic Plans using Transformerscited by 41
  23. 2023TRAM: Benchmarking Temporal Reasoning for Large Language Modelscited by 40
  24. 2024Graph-enhanced Large Language Models in Asynchronous Plan Reasoningcited by 37

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 16; 2026: 16.

32 papers.

Sort:

By area:

  1. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 149
  2. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 83
  3. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 50
  4. 2026The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Breakcited by 25
  5. 2025Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMscited by 24
  6. 2025COMPASS: Enhancing Agent Long-Horizon Reasoning with Evolving Contextcited by 20
  7. 2025A Survey on Large Language Models for Automated Planningcited by 17
  8. 2026Agentic Artificial Intelligence (AI): Architectures, Taxonomies, and Evaluation of Large Language Model Agentscited by 14
  9. 2025PLANET: A Collection of Benchmarks for Evaluating LLMs' Planning Capabilitiescited by 7
  10. 2026PM-Bench: Evaluating Prospective Memory in LLM Agentscited by 7
  11. 2025TextAtari: 100K Frames Game Playing with Language Agentscited by 6
  12. 2025HeroBench: A Benchmark for Long-Horizon Planning and Structured Reasoning in Virtual Worldscited by 6
20 more
  1. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 5
  2. 2026AsyncTool: Evaluating the Asynchronous Function Calling Capability under Multi-Task Scenarioscited by 5
  3. 2025Reflex First, Reflect Later: Latency-Aware Embodied LLM Agents for Dynamic Responsecited by 4
  4. 2025Skill-Aligned Fairness in Multi-Agent Learning for Collaboration in Healthcarecited by 3
  5. 2025ATTS: Asynchronous Test-Time Scaling via Conformal Predictioncited by 3
  6. 2025ParaCook: On Time-Efficient Planning for Multi-Agent Systemscited by 3
  7. 2026Embodied Task Planning via Graph-Informed Action Generation with Large Language Modelscited by 2
  8. 2026TabAgent: A Framework for Replacing Agentic Generative Components with Tabular-Textual Classifierscited by 2
  9. 2026Engagement Process: Rethinking the Temporal Interface of Action and Observationcited by 2
  10. 2025The Hunger Game Debate: On the Emergence of Over-Competition in Multi-Agent Systemscited by 1
  11. 2026PACE: Adaptive Budget Allocation for Time-Efficient Embodied Planningcited by 1
  12. 2025AdaFair-MARL: Enforcing Adaptive Fairness Constraints in Multi-Agent Reinforcement Learningcited by 0
  13. 2026Latent Action Reparameterization for Efficient Agent Inferencecited by 0
  14. 2026Robust Asynchronous Planning via Auto-Formalizationcited by 0
  15. 2026Searching for Synergy in Shared Workspace Human-AI Collaborationcited by 0
  16. 2026Beyond Sequential Interaction: Benchmarking Parallel Execution and Coordination for GUI Agentscited by 0
  17. 2026HaReCAP: Habitual-action Grounding for Recursive Large Language Model Agentscited by 0
  18. 2026LLMs are General Asynchronous Agentscited by 0
  19. 2026ReLiveGym: Evaluating Long-Lived Agents over Weeks of Replayed Realitycited by 0
  20. 2026ASIWorld: Benchmarking Autonomous Agents in a Persistent Survival Sandboxcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning When to Plan: Efficiently Allocating Test-Time Compute for LLM Agents
  2. 2026Why Reasoning Fails to Plan: A Planning-Centric Analysis of Long-Horizon Decision Making in LLM Agents
  3. 2024Understanding the planning of LLM agents: A survey
  4. 2026Planner Matters! An Efficient and Unbalanced Multi-agent Collaboration Framework for Long-horizon Planning
  5. 2026Agentic LLM Planning via Step-Wise PDDL Simulation: An Empirical Characterisation
  6. 2024Query-Efficient Planning with Language Models
  7. 2024Can We Rely on LLM Agents to Draft Long-Horizon Plans? Let's Take TravelPlanner as an Example
  8. 2023On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)