mlsanityData to 7 Oct 2026

The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AI

arXiv 2103.14025 ↗First posted 25 Mar 2021cs.CV cs.AI cs.LGcited by 64

Not on the map. Why?

Abstract (arXiv)

We introduce a visually-guided and physics-driven task-and-motion planning benchmark, which we call the ThreeDWorld Transport Challenge. In this challenge, an embodied agent equipped with two 9-DOF articulated arms is spawned randomly in a simulated physical home environment. The agent is required to find a small set of objects scattered around the house, pick them up, and transport them to a desired final location. We also position containers around the house that can be used as tools to assist with transporting objects efficiently. To complete the task, an embodied agent must plan a sequence of actions to change the state of a large number of objects in the face of realistic physical constraints. We build this benchmark challenge using the ThreeDWorld simulation: a virtual 3D environment where all objects respond to physics, and where can be controlled using fully physics-driven navigation and interaction API. We evaluate several existing agents on this benchmark. Experimental results suggest that: 1) a pure RL model struggles on this challenge; 2) hierarchical planning-based agents can transport some objects but still far from solving this task. We anticipate that this benchmark will empower researchers to develop more intelligent physics-driven robots for the physical world.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Builds on 38 of its 53 references that are in the map. First posted on arXiv 25 Mar 2021.

Builds on

38 of 53 references.

References · 38

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  3. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  4. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  5. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  6. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  7. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  8. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  9. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  10. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  11. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  12. 2016Semantic Scene Completion from a Single Depth Imagecited by 468
26 more
  1. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  2. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  3. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 351
  4. 2017Embodied Question Answeringcited by 335
  5. 2020Learning to Explore using Active Neural SLAMcited by 308
  6. 2020Integrated Task and Motion Planningcited by 281
  7. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 249
  8. 2017Cognitive Mapping and Planning for Visual Navigationcited by 209
  9. 2018Semi-parametric Topological Memory for Navigationcited by 203
  10. 2016Learning to Navigate in Complex Environmentscited by 192
  11. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  12. 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
  13. 2020Rearrangement: A Challenge for Embodied AIcited by 153
  14. 2020Watch-And-Help: A Challenge for Social Perception and Human-AI Collaborationcited by 131
  15. 2019Learning Exploration Policies for Navigationcited by 113
  16. 2018Building Generalizable Agents with a Realistic and Rich 3D Environmentcited by 109
  17. 2020Occupancy Anticipation for Efficient Exploration and Navigationcited by 104
  18. 2019Look, Listen, and Act: Towards Audio-Visual Embodied Navigationcited by 95
  19. 2019Interactive Gibson Benchmark (iGibson 0.5): A Benchmark for Interactive Navigation in Cluttered Environmentscited by 89
  20. 2018Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigationcited by 84
  21. 2016DeepMind Labcited by 74
  22. 2019Combining Optimal Control and Learning for Visual Navigation in Novel Environmentscited by 50
  23. 2020ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulationcited by 47
  24. 2018Active model learning and diverse action sampling for task and motion planningcited by 28
  25. 2018Beauty and the Beast: Optimal Methods Meet Learning for Drone Racingcited by 20
  26. 2019The Animal-AI Environment: Training and Testing Animal-Like Artificial Cognitioncited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 11; 2023: 15; 2024: 15; 2025: 8; 2026: 8.

64 papers.

Sort:

By area:

  1. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  2. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  3. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  4. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 313
  5. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 231
  6. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  7. 2021iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Taskscited by 191
  8. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 161
  9. 2024BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulationcited by 140
  10. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  11. 2024SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agentscited by 102
  12. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 101
38 more
  1. 2021PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physicscited by 100
  2. 2022Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigationcited by 86
  3. 2023ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenescited by 68
  4. 2023A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Modelscited by 61
  5. 2022TASKOGRAPHY: Evaluating robot task planning over large 3D scene graphscited by 59
  6. 2021Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Languagecited by 55
  7. 2024GRUtopia: Dream General Robots in a City at Scalecited by 54
  8. 2023SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real Worldcited by 47
  9. 2022Multi-skill Mobile Manipulation for Object Rearrangementcited by 46
  10. 2022Retrospectives on the Embodied AI Workshopcited by 46
  11. 2021Curious Representation Learning for Embodied Intelligencecited by 43
  12. 2023Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Modelscited by 37
  13. 2024LLaMAR: Long-Horizon Planning for Multi-Agent Robots in Partially Observable Environmentscited by 32
  14. 2022TIDEE: Tidying Up Novel Rooms using Visuo-Semantic Commonsense Priorscited by 30
  15. 2025VideoGameBench: Can Vision-Language Models complete popular video games?cited by 29
  16. 2023Mini-BEHAVIOR: A Procedurally Generated Benchmark for Long-horizon Decision-Making in Embodied AIcited by 23
  17. 2024PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videoscited by 21
  18. 2022Learning Active Camera for Multi-Object Navigationcited by 20
  19. 2024ManiSkill-HAB: A Benchmark for Low-Level Manipulation in Home Rearrangement Taskscited by 20
  20. 2024Hazards in Daily Life? Enabling Robots to Proactively Detect and Resolve Anomaliescited by 19
  21. 2023Harmonic Mobile Manipulationcited by 16
  22. 2022N2M2: Learning Navigation for Arbitrary Mobile Manipulation Motions in Unseen and Dynamic Environmentscited by 15
  23. 2024Why Not Use Your Textbook? Knowledge-Enhanced Procedure Planning of Instructional Videoscited by 15
  24. 2022Finding Fallen Objects Via Asynchronous Audio-Visual Integrationcited by 13
  25. 2024Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challengecited by 13
  26. 2024CaPo: Cooperative Plan Optimization for Efficient Embodied Multi-Agent Cooperationcited by 12
  27. 2025EcomScriptBench: A Multi-task Benchmark for E-commerce Script Planning via Step-wise Intention-Driven Product Associationcited by 10
  28. 2023SoftMAC: Differentiable Soft Body Simulation with Forecast-based Contact Model and Two-way Coupling with Articulated Rigid Bodies and Clothescited by 9
  29. 2025MORE: Mobile Manipulation Rearrangement Through Grounded Language Reasoningcited by 9
  30. 2023Planning as In-Painting: A Diffusion-Based Embodied Task Planning Framework for Environments under Uncertaintycited by 8
  31. 2024HELPER-X: A Unified Instructable Embodied Agent to Tackle Four Interactive Vision-Language Domains with Memory-Augmented Language Modelscited by 8
  32. 2024EscapeBench: Towards Advancing Creative Intelligence of Language Model Agentscited by 5
  33. 2021OPEn: An Open-ended Physics Environment for Learning Without a Taskcited by 4
  34. 2024WorldAPIs: The World Is Worth How Many APIs? A Thought Experimentcited by 4
  35. 2025Learn as Individuals, Evolve as a Team: Multi-agent LLMs Adaptation in Embodied Environmentscited by 4
  36. 2023When Learning Is Out of Reach, Reset: Generalization in Autonomous Visuomotor Reinforcement Learningcited by 3
  37. 2023Moving Forward by Moving Backward: Embedding Action Impact over Action Semanticscited by 3
  38. 2025EmboMatrix: A Scalable Training-Ground for Embodied Decision-Makingcited by 3

50 most cited of 64 All 64 (JSON)