mlsanityData to 7 Oct 2026

ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation

arXiv 2007.04954 ↗First posted 9 Jul 2020cs.CV cs.GR cs.LGcited by 249

On the map: World models → Video-generation world models and simulators

Abstract (arXiv)

We introduce ThreeDWorld (TDW), a platform for interactive multi-modal physical simulation. TDW enables simulation of high-fidelity sensory data and physical interactions between mobile agents and objects in rich 3D environments. Unique properties include: real-time near-photo-realistic image rendering; a library of objects and environments, and routines for their customization; generative procedures for efficiently building classes of new environments; high-fidelity audio rendering; realistic physical interactions for a variety of material types, including cloths, liquid, and deformable objects; customizable agents that embody AI agents; and support for human interactions with VR devices. TDW's API enables multiple agents to interact within a simulation and returns a range of sensor and physics data representing the state of the world. We present initial experiments enabled by TDW in emerging research directions in computer vision, machine learning, and cognitive science, including multi-modal physical scene understanding, physical dynamics predictions, multi-agent interactions, models that learn like a child, and attention studies in humans and neural networks.

Key facts. Cited by 249 papers in the map (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 4 of its 58 references that are in the map. First posted on arXiv 9 Jul 2020.

Builds on

4 of 58 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017AI2-THOR: An Interactive 3D Environment for Visual AI14 Dec

Other references · 3

  1. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  2. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  3. 2016Learning to Poke by Poking: Experiential Learning of Intuitive Physicscited by 152

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 33; 2022: 47; 2023: 61; 2024: 47; 2025: 34; 2026: 26.

249 papers.

Sort:

By area:

  1. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  2. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  3. 2022Kubric: A scalable dataset generatorcited by 373
  4. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  5. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 326
  6. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 313
  7. 2023Tool Learning with Foundation Modelscited by 302
  8. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  9. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 257
  10. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 256
  11. 2023Orbit: A Unified Simulation Framework for Interactive Robot Learning Environmentscited by 239
  12. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 231
38 more
  1. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 220
  2. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  3. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  4. 2021iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Taskscited by 191
  5. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  6. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  7. 2020iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenescited by 151
  8. 2021Physion: Evaluating Physical Prediction from Vision in Humans and Machinescited by 145
  9. 2023AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation Systemcited by 136
  10. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  11. 2023HomeRobot: Open-Vocabulary Mobile Manipulationcited by 104
  12. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 101
  13. 2021PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physicscited by 100
  14. 2021ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentationcited by 93
  15. 2022SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learningcited by 93
  16. 2023Improving Multimodal Datasets with Image Captioningcited by 93
  17. 2022Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigationcited by 86
  18. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  19. 2021Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoningcited by 68
  20. 2023ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenescited by 68
  21. 2021ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representationscited by 67
  22. 2022Learning Neural Acoustic Fieldscited by 65
  23. 2023LLM-Coordination: Evaluating and Analyzing Multi-agent Coordination Abilities in Large Language Modelscited by 65
  24. 2025Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generationcited by 65
  25. 2021The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AIcited by 64
  26. 2022DialFRED: Dialogue-Enabled Agents for Embodied Instruction Followingcited by 63
  27. 2023Gen2Sim: Scaling up Robot Learning in Simulation with Generative Modelscited by 62
  28. 2022See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulationcited by 61
  29. 2023A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Modelscited by 61
  30. 2021Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulationcited by 59
  31. 2022TASKOGRAPHY: Evaluating robot task planning over large 3D scene graphscited by 59
  32. 2022Housekeep: Tidying Virtual Households using Commonsense Reasoningcited by 57
  33. 2021Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Languagecited by 55
  34. 2024PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigatorscited by 55
  35. 2023How To Not Train Your Dragon: Training-free Embodied Object Goal Navigation with Semantic Frontierscited by 53
  36. 2023Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Propertiescited by 53
  37. 2022Exploration via Elliptical Episodic Bonusescited by 51
  38. 2023SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real Worldcited by 47

50 most cited of 249 All 249 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LatticeWorld: A Multimodal Large Language Model-Empowered Framework for Interactive Complex World Generation
  2. 20263D Generation for Embodied AI and Robotic Simulation: A Survey
  3. 2026SimWorlds: A Multi-Agent System for Dynamic 3D Scene Creation
  4. 2026Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents
  5. 2025Simulating the Visual World with Artificial Intelligence: A Roadmap
  6. 2026GS-Agent: Creating 4D Physical Worlds With Generative Simulation
  7. 20264DSynth: Controllable Procedural World Synthesis for Dynamic Embodied Simulation
  8. 20254DWorldBench: A Comprehensive Evaluation Framework for 3D/4D World Generation Models