mlsanityData to 14 Oct 2026

PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks

arXiv 2411.00081 ↗First posted 31 Oct 2024cs.RO cs.AIcited by 82

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

We present a benchmark for Planning And Reasoning Tasks in humaN-Robot collaboration (PARTNR) designed to study human-robot coordination in household activities. PARTNR tasks exhibit characteristics of everyday tasks, such as spatial, temporal, and heterogeneous agent capability constraints. We employ a semi-automated task generation pipeline using Large Language Models (LLMs), incorporating simulation in the loop for grounding and verification. PARTNR stands as the largest benchmark of its kind, comprising 100,000 natural language tasks, spanning 60 houses and 5,819 unique objects. We analyze state-of-the-art LLMs on PARTNR tasks, across the axes of planning, perception and skill execution. The analysis reveals significant limitations in SoTA models, such as poor coordination and failures in task tracking and recovery from errors. When LLMs are paired with real humans, they require 1.5x as many steps as two humans collaborating and 1.1x more steps than a single human, underscoring the potential for improvement in these models. We further show that fine-tuning smaller LLMs with planning data can achieve performance on par with models 9 times larger, while being 8.6x faster at inference. Overall, PARTNR highlights significant challenges facing collaborative embodied agents and aims to drive research in this direction.

Key facts. Cited by 82 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 0 of its 64 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

0 of 64 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 30; 2026: 47.

82 papers.

Sort:

By area:

  1. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 221
  2. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 83
  3. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 40
  4. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  5. 2025Rethinking the Embodied Gap in Vision-and-Language Navigation: A Holistic Study of Physical and Visual Disparitiescited by 25
  6. 2025CognitiveDrone: A VLA Model and Evaluation Benchmark for Real-Time Cognitive Task Solving and Reasoning in UAVscited by 22
  7. 2024TeamCraft: A Benchmark for Multi-Modal Multi-Agent Systems in Minecraftcited by 21
  8. 2025Collaborating Action by Action: A Multi-agent LLM Framework for Embodied Reasoningcited by 21
  9. 2025Embodied Agents Meet Personalization: Investigating Challenges and Solutions Through the Lens of Memory Utilizationcited by 19
  10. 2025VIKI-R: Coordinating Embodied Multi-Agent Cooperation via Reinforcement Learningcited by 18
  11. 2025EgoNormia: Benchmarking Physical Social Norm Understandingcited by 13
  12. 2026The Five Ws of Multi-Agent Communication: Who Talks to Whom, When, What, and Why -- A Survey from MARL to Emergent Language and LLMscited by 10
38 more
  1. 2025A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomycited by 9
  2. 2025Orchestrating Human-AI Teams: The Manager Agent as a Unifying Research Challengecited by 8
  3. 2025ViGiL3D: A Linguistically Diverse Dataset for 3D Visual Groundingcited by 7
  4. 2025PLANET: A Collection of Benchmarks for Evaluating LLMs' Planning Capabilitiescited by 7
  5. 2025CREW-WILDFIRE: Benchmarking Agentic Multi-Agent Collaborations at Scalecited by 7
  6. 2025Modeling Others' Minds as Codecited by 7
  7. 2026RoboAgent: Chaining Basic Capabilities for Embodied Task Planningcited by 7
  8. 2026AdaPlanBench: Evaluating Adaptive Planning in Large Language Model Agents under World and User Constraintscited by 7
  9. 2024MageBench: Bridging Large Multimodal Models to Agentscited by 6
  10. 2025ADAPT: Actively Discovering and Adapting to Preferences for any Taskcited by 6
  11. 2025TextAtari: 100K Frames Game Playing with Language Agentscited by 6
  12. 2025When Should Users Check? Modeling Confirmation Frequency in Multi-Step Agentic AI Taskscited by 6
  13. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 6
  14. 2025Agent Identity Evals: Measuring Agentic Identitycited by 5
  15. 2025IndustryNav: Exploring Spatial Reasoning of Embodied Agents in Dynamic Industrial Navigationcited by 5
  16. 2026MA-EgoQA: Question Answering over Egocentric Videos from Multiple Embodied Agentscited by 5
  17. 2026WorldLines: Benchmarking and Modeling Long-Horizon Stateful Embodied Agentscited by 5
  18. 2025Plan Verification for LLM-Based Embodied Task Completion Agentscited by 4
  19. 2026Seeing Together: Multi-Robot Cooperative Egocentric Spatial Reasoning with Multimodal Large Language Modelscited by 4
  20. 2024λ: A Benchmark for Data-Efficiency in Long-Horizon Indoor Mobile Manipulation Roboticscited by 3
  21. 2025EmbodiedAgent: A Scalable Hierarchical Approach to Overcome Practical Challenge in Multi-Robot Controlcited by 3
  22. 2025Embodied AI with Foundation Models for Mobile Service Robots: A Systematic Reviewcited by 3
  23. 2026Game-Theoretic Lens on LLM-based Multi-Agent Systemscited by 3
  24. 2026EnactToM: An Evolving Benchmark for Functional Theory of Mind in Embodied Agentscited by 3
  25. 2026MECoBench: A Systematic Study of Multimodal Agent Collaboration in Embodied Environmentscited by 3
  26. 2025Exploring 3D Reasoning-Driven Planning: From Implicit Human Intentions to Route-Aware Activity Planningcited by 2
  27. 2025GroundAct: Can LLM Agents Ground Actions in Environmental States?cited by 2
  28. 2025GauDP: Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policiescited by 2
  29. 2026EmbodiedGovBench: A Benchmark for Governance, Recovery, and Upgrade Safety in Embodied Agent Systemscited by 2
  30. 2026PersonalHomeBench: Evaluating Agents in Personalized Smart Homescited by 2
  31. 2026LLawCo: Learning Laws of Cooperation for Modeling Embodied Multi-Agent Behaviorcited by 2
  32. 2026HumanCLAW: Can Vision-Language Models Act Through a Body?cited by 2
  33. 2026DeliveryGym: An RL Environment for Long-Horizon Embodied Agent Planning with Adaptive Curriculumcited by 2
  34. 2026MAGMA-GEN: Validated Recovery Supervision from Ambiguous Failures via Counterfactual Re-Executioncited by 2
  35. 2024Hierarchical Object-Oriented POMDP Planning for Object Rearrangementcited by 1
  36. 2025Active Inference AI Systems for Scientific Discoverycited by 1
  37. 2025Evaluation of Habitat Robotics using Large Language Modelscited by 1
  38. 2025ProToM: Promoting Prosocial Behaviour via Theory of Mind-Informed Feedbackcited by 1

50 most cited of 82 All 82 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LoTa-Bench: Benchmarking Language-oriented Task Planners for Embodied Agents
  2. 2022Inner Monologue: Embodied Reasoning through Planning with Language Models
  3. 2024LLaMAR: Long-Horizon Planning for Multi-Agent Robots in Partially Observable Environments
  4. 2026STEP: State-Aware Task Estimation and Planning with Multi-Modal LLMs for Human-Robot Collaboration
  5. 2024Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration
  6. 2026From Assumptions to Actions: Turning LLM Reasoning into Uncertainty-Aware Planning for Embodied Agents
  7. 2025HELP: Hierarchical Embodied Language Planner for Household Tasks
  8. 2025Constrained Natural Language Action Planning for Resilient Embodied Systems