# ET-Plan-Bench: Embodied Task-level Planning Benchmark Towards Spatial-Temporal Cognition with Foundation Models

arXiv: https://arxiv.org/abs/2410.14682 · HTML: https://mlsanity.com/paper/2410.14682/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 2 Oct 2024.

## Builds on (most cited first, 30 of 30)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [Faith and Fate: Limits of Transformers on Compositionality](https://mlsanity.com/paper/2305.18654/) (2023)
- [VirtualHome: Simulating Household Activities via Programs](https://mlsanity.com/paper/1806.07011/) (2018)
- [ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills](https://mlsanity.com/paper/2302.04659/) (2023)
- [On the Planning Abilities of Large Language Models : A Critical Investigation](https://mlsanity.com/paper/2305.15771/) (2023)
- [EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought](https://mlsanity.com/paper/2305.15021/) (2023)
- [Large Language Models as Commonsense Knowledge for Large-Scale Task Planning](https://mlsanity.com/paper/2305.14078/) (2023)
- [RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation](https://mlsanity.com/paper/2311.01455/) (2023)
- [Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation](https://mlsanity.com/paper/2306.11290/) (2023)
- [BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments](https://mlsanity.com/paper/2108.03332/) (2021)
- [Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration](https://mlsanity.com/paper/2010.09890/) (2020)
- [EgoTaskQA: Understanding Human Tasks in Egocentric Videos](https://mlsanity.com/paper/2210.03929/) (2022)
- [EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models](https://mlsanity.com/paper/2311.15596/) (2023)
- [LoTa-Bench: Benchmarking Language-oriented Task Planners for Embodied Agents](https://mlsanity.com/paper/2402.08178/) (2024)
- [Mini-BEHAVIOR: A Procedurally Generated Benchmark for Long-horizon Decision-Making in Embodied AI](https://mlsanity.com/paper/2310.01824/) (2023)
- [HandMeThat: Human-Robot Communication in Physical and Social Environments](https://mlsanity.com/paper/2310.03779/) (2023)

## Built on by (most cited first, 13 of 13)

- [ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulation](https://mlsanity.com/paper/2505.09698/) (2025)
- [Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey](https://mlsanity.com/paper/2506.11102/) (2025)
- [Towards foundational LiDAR world models with efficient latent flow matching](https://mlsanity.com/paper/2506.23434/) (2025)
- [DeliveryBench: Can Agents Earn Profit in Real World?](https://mlsanity.com/paper/2512.19234/) (2025)
- [PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments](https://mlsanity.com/paper/2510.21111/) (2025)
- [AsgardBench -- Evaluating Visually Grounded Interactive Planning Under Minimal Feedback](https://mlsanity.com/paper/2603.15888/) (2026)
- [SpatialAct: Probing Spatial Reasoning-to-Action Capabilities of VLM Agents in 3D Scenes](https://mlsanity.com/paper/2605.31148/) (2026)
- [Evaluating Time Awareness and Cross-modal Active Perception of Large Models via 4D Escape Room Task](https://mlsanity.com/paper/2603.15467/) (2026)
- [UniETP: Unifying Environments for Generalizable Embodied Task Planning](https://mlsanity.com/paper/2607.18062/) (2026)
- [NutriBench-Kitchen: Benchmarking Embodied AI for Nutrition Management](https://mlsanity.com/paper/2609.07135/) (2026)
- [CLAMP: Constrained Decoding for Vision-Language Embodied Planning](https://mlsanity.com/paper/2609.08602/) (2026)
- [EMBER-Bench: Benchmarking Cross-Event Causal Memory in Long-Horizon Embodied Tasks](https://mlsanity.com/paper/2610.05013/) (2026)
- [ABot-ER: Unified Embodied Reasoning through Agentic Data-Model Co-Evolution](https://mlsanity.com/paper/2610.18355/) (2026)

## Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) have spurred numerous attempts to apply these technologies to embodied tasks, particularly focusing on high-level task planning and task decomposition. To further explore this area, we introduce a new embodied task planning benchmark, ET-Plan-Bench, which specifically targets embodied task planning using LLMs. It features a controllable and diverse set of embodied tasks varying in different levels of difficulties and complexities, and is designed to evaluate two critical dimensions of LLMs' application in embodied task understanding: spatial (relation constraint, occlusion for target objects) and temporal & causal understanding of the sequence of actions in the environment. By using multi-source simulators as the backend simulator, it can provide immediate environment feedback to LLMs, which enables LLMs to interact dynamically with the environment and re-plan as necessary. We evaluated the state-of-the-art open source and closed source foundation models, including GPT-4, LLAMA and Mistral on our proposed benchmark. While they perform adequately well on simple navigation tasks, their performance can significantly deteriorate when faced with tasks that require a deeper understanding of spatial, temporal, and causal relationships. Thus, our benchmark distinguishes itself as a large-scale, quantifiable, highly automated, and fine-grained diagnostic framework that presents a significant challenge to the latest foundation models. We hope it can spark and drive further research in embodied task planning using foundation models.