mlsanityData to 7 Oct 2026

LoTa-Bench: Benchmarking Language-oriented Task Planners for Embodied Agents

arXiv 2402.08178 ↗First posted 13 Feb 2024cs.AIcited by 51

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Large language models (LLMs) have recently received considerable attention as alternative solutions for task planning. However, comparing the performance of language-oriented task planners becomes difficult, and there exists a dearth of detailed exploration regarding the effects of various factors such as pre-trained model selection and prompt construction. To address this, we propose a benchmark system for automatically quantifying performance of task planning for home-service embodied agents. Task planners are tested on two pairs of datasets and simulators: 1) ALFRED and AI2-THOR, 2) an extension of Watch-And-Help and VirtualHome. Using the proposed benchmark system, we perform extensive experiments with LLMs and prompts, and explore several enhancements of the baseline planner. We expect that the proposed benchmark tool would accelerate the development of language-oriented task planners.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 33 of its 55 references that are in the map. First posted on arXiv 13 Feb 2024.

Builds on

33 of 55 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agents3 May
  2. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents18 Jan

Other references · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  9. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  10. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  11. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  12. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
19 more
  1. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  2. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  3. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  4. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  5. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  6. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  7. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  8. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 256
  9. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  10. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  11. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  12. 2018PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planningcited by 147
  13. 2020Watch-And-Help: A Challenge for Social Perception and Human-AI Collaborationcited by 131
  14. 2019Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generationcited by 83
  15. 2022Learning Neuro-Symbolic Skills for Bilevel Planningcited by 55
  16. 2022On Grounded Planning for Embodied Tasks with Language Modelscited by 33
  17. 2019Regression Planning Networkscited by 32
  18. 2020Broadly-Exploring, Local-Policy Trees for Long-Horizon Task Planningcited by 26
  19. 2021Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoningcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 22; 2026: 22.

51 papers.

Sort:

By area:

  1. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  2. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  3. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  4. 2024Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Makingcited by 142
  5. 2024SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agentscited by 102
  6. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  7. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  8. 2024KARMA: Augmenting Embodied AI Agents with Long-and-short Term Memory Systemscited by 38
  9. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  10. 2025World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planningcited by 26
  11. 2025ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulationcited by 24
  12. 2025WorldPrediction: A Benchmark for High-level World Modeling and Long-horizon Procedural Planningcited by 19
38 more
  1. 2024ET-Plan-Bench: Embodied Task-level Planning Benchmark Towards Spatial-Temporal Cognition with Foundation Modelscited by 11
  2. 2025ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learningcited by 10
  3. 2025ReAcTree: Hierarchical LLM Agent Trees with Control Flow for Long-Horizon Task Planningcited by 10
  4. 2025Safety of Embodied Navigation: A Surveycited by 9
  5. 2026VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agentscited by 9
  6. 2026AIR: Improving Agent Safety through Incident Responsecited by 8
  7. 2024Socratic Planner: Self-QA-Based Zero-Shot Planning for Embodied Instruction Followingcited by 7
  8. 2025REI-Bench: Can Embodied Agents Understand Vague Human Instructions in Task Planning?cited by 7
  9. 2025BEAT: Visual Backdoor Attacks on VLM-based Embodied Agents via Contrastive Trigger Learningcited by 7
  10. 2026Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agentscited by 7
  11. 2026PLanAR: Planning-Language-Grounded Agentic Reasoning for Robot Manipulationcited by 6
  12. 2026RoboAgent: Chaining Basic Capabilities for Embodied Task Planningcited by 6
  13. 2025IndustryNav: Exploring Spatial Reasoning of Embodied Agents in Dynamic Industrial Navigationcited by 5
  14. 2025ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Makingcited by 4
  15. 2025Mind and Motion Aligned: A Joint Evaluation IsaacSim Benchmark for Task Planning and Low-Level Policies in Mobile Manipulationcited by 4
  16. 2025MADRA: Multi-Agent Debate for Risk-Aware Embodied Planningcited by 4
  17. 2026SAFE-Pruner: Semantic Attention-Guided Future-Aware Token Pruning for Efficient Vision-Language-Action Manipulationcited by 4
  18. 2025Embodied AI: Emerging Risks and Opportunities for Policy Actioncited by 3
  19. 2025AURA: A Diagnostic Framework for Tracking User Satisfaction of Interactive Planning Agentscited by 2
  20. 2025Grounding Language Models with Semantic Digital Twins for Robotic Planningcited by 2
  21. 2026How Foundational Skills Influence VLM-based Embodied Agents:A Native Perspectivecited by 2
  22. 2025SDA-PLANNER: State-Dependency Aware Adaptive Planner for Embodied Task Planningcited by 1
  23. 2025City Navigation in the Wild: Exploring Emergent Navigation from Web-Scale Knowledge in MLLMscited by 1
  24. 2026When Robots Do the Chores: A Benchmark and Agent for Long-Horizon Household Task Executioncited by 1
  25. 2026HumanCLAW: Can Vision-Language Models Act Through a Body?cited by 1
  26. 2025RoboTidy : A 3D Gaussian Splatting Household Tidying Benchmark for Embodied Navigation and Actioncited by 0
  27. 2026AsgardBench -- Evaluating Visually Grounded Interactive Planning Under Minimal Feedbackcited by 0
  28. 2026Environmental Understanding Vision-Language Model for Embodied Agentcited by 0
  29. 2026NEXUS: Continual Learning of Symbolic Constraints for Safe and Robust Embodied Planningcited by 0
  30. 2026TaskGround: Structured Executable Task Inference for Full-Scene Household Reasoningcited by 0
  31. 2026A Practical Recipe Towards Improving Sim-and-Real Correlation for VLA Evaluationcited by 0
  32. 2026SIMMER: Benchmarking Latent Failures in LLM Executable Planning with a World Modelcited by 0
  33. 2026UniETP: Unifying Environments for Generalizable Embodied Task Planningcited by 0
  34. 2026MetaSpace: Metamorphic Testing for Spatial Cognition in Embodied Agentscited by 0
  35. 2026GraphThink: Graph-Enhanced LLM Thinking for Long-Horizon Embodied Task Planningcited by 0
  36. 2026Meta-Ctrl: Guaranteed Plan Generation by Decoupling Syntactic and Semantic Constraintscited by 0
  37. 2026Do World Models Make Better Robots? A Survey of Evaluation Benchmarks for Predictive Embodied Intelligencecited by 0
  38. 2026PlanGuard: A Guardrail for Multi-Step Plan Safety in Embodied Agentscited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025HELP: Hierarchical Embodied Language Planner for Household Tasks
  2. 2024PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks
  3. 2024ActPlan-1K: Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activities
  4. 2026Structured Self-Consistency:A Multi-Task Evaluation of LLMs on VirtualHome
  5. 2026CoCoBench: A Cooperative Coordination Benchmark for Embodied Multi-Agent Task Planning
  6. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models
  7. 2025Plan Verification for LLM-Based Embodied Task Completion Agents
  8. 2025Using Language Models as Closed-Loop High-Level Planners for Robotics Applications: A Brief Overview and Benchmarks