mlsanityData to 14 Oct 2026

Learning Interactive Real-World Simulators

arXiv 2310.06114 ↗First posted 9 Oct 2023cs.AIcited by 458

On the map: World models → Video-generation world models and simulators

Abstract (arXiv)

Generative models trained on internet data have revolutionized how text, image, and video content can be created. Perhaps the next milestone for generative models is to simulate realistic experience in response to actions taken by humans, robots, and other interactive agents. Applications of a real-world simulator range from controllable content creation in games and movies, to training embodied agents purely in simulation that can be directly deployed in the real world. We explore the possibility of learning a universal simulator (UniSim) of real-world interaction through generative modeling. We first make the important observation that natural datasets available for learning a real-world simulator are often rich along different dimensions (e.g., abundant objects in image data, densely sampled actions in robotics data, and diverse movements in navigation data). With careful orchestration of diverse datasets, each providing a different aspect of the overall experience, we can simulate the visual outcome of both high-level instructions such as "open the drawer" and low-level controls from otherwise static scenes and objects. We use the simulator to train both high-level vision-language policies and low-level reinforcement learning policies, each of which can be deployed in the real world in zero shot after training purely in simulation. We also show that other types of intelligence such as video captioning models can benefit from training with simulated experience, opening up even wider applications. Video demos can be found at https://universal-simulator.github.io.

Key facts. Cited by 458 papers in the map (data through 14 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 72 of its 95 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

72 of 95 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Probabilistic Adaptation of Text-to-Video Models2 Jun
  2. 2020Denoising Diffusion Probabilistic Models19 Jun
  3. 2017Matterport3D: Learning from RGB-D Data in Indoor Environments18 Sep

Other references · 69

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2022Classifier-Free Diffusion Guidancecited by 5,847
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  9. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  10. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  11. 2022Video Diffusion Modelscited by 2,305
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
38 more
  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  2. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  3. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  4. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  5. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  6. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,476
  7. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  8. 2023Mastering Diverse Domains through World Modelscited by 1,253
  9. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  10. 2023PaLM 2 Technical Reportcited by 1,162
  11. 2017Hindsight Experience Replaycited by 1,136
  12. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  13. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  14. 2020Mastering Atari with Discrete World Modelscited by 1,007
  15. 2021Scaling Vision Transformerscited by 999
  16. 2018DeepMind Control Suitecited by 956
  17. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  18. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  19. 2017Dense-Captioning Events in Videoscited by 793
  20. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  21. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  22. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
  23. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  24. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  25. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  26. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 465
  27. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  28. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  29. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 413
  30. 2022Transformers are Sample-Efficient World Modelscited by 328
  31. 2022UL2: Unifying Language Learning Paradigmscited by 295
  32. 2019Challenges of Real-World Reinforcement Learningcited by 291
  33. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 270
  34. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 265
  35. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 256
  36. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  37. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 250
  38. 2022Interactive Language: Talking to Robots in Real Timecited by 218

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 111; 2025: 161; 2026: 180.

458 papers.

Sort:

By area:

  1. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  2. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  3. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  4. 2024Genie: Generative Interactive Environmentscited by 747
  5. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 625
  6. 2026World Action Models are Zero-shot Policiescited by 465
  7. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 451
  8. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  9. 2024Latent Action Pretraining from Videoscited by 321
  10. 2024Diffusion Models Are Real-Time Game Enginescited by 296
  11. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 296
  12. 2024Diffusion Policy Policy Optimizationcited by 280
38 more
  1. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  2. 2024Navigation World Modelscited by 272
  3. 2025Motus: A Unified Latent Action World Modelcited by 251
  4. 2023Any-point Trajectory Modeling for Policy Learningcited by 215
  5. 2024How Far is Video Generation from World Model: A Physical Law Perspectivecited by 212
  6. 2025EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Videocited by 211
  7. 2025Imagine while Reasoning in Space: Multimodal Visualization-of-Thoughtcited by 206
  8. 2024RoboDreamer: Learning Compositional World Models for Robot Imaginationcited by 195
  9. 2025Matrix-game 2.0: An open-source real-time and streaming interactive world modelcited by 186
  10. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 179
  11. 2025WorldMem: Long-term Consistent World Simulation with Memorycited by 168
  12. 2025Ctrl-World: A Controllable Generative World Model for Robot Manipulationcited by 167
  13. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  14. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 165
  15. 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrievalcited by 163
  16. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  17. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 146
  18. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  19. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 135
  20. 2025Unified Vision-Language-Action Modelcited by 134
  21. 2023Large Language Models for Robotics: A Surveycited by 133
  22. 2025GameFactory: Creating New Games with Generative Interactive Videoscited by 130
  23. 2024GameGen-X: Interactive Open-world Game Video Generationcited by 128
  24. 2024Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigationcited by 126
  25. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 124
  26. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 114
  27. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  28. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  29. 2024IRASim: A Fine-Grained World Model for Robot Manipulationcited by 100
  30. 2025MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraftcited by 97
  31. 2025TesserAct: Learning 4D Embodied World Modelscited by 94
  32. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  33. 2024Video as the New Language for Real-World Decision Makingcited by 87
  34. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 86
  35. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 82
  36. 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 80
  37. 2024PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generationcited by 80
  38. 2024The Matrix: Infinite-Horizon World Generation with Real-Time Moving Controlcited by 80

50 most cited of 458 All 458 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Simulating the Real World: A Unified Survey of Multimodal Generative Models
  2. 2026Interactive World Simulator for Robot Policy Training and Evaluation
  3. 2026Scaling Sim-to-Real VLA Reinforcement Learning with Generative 3D Worlds
  4. 20263D Generation for Embodied AI and Robotic Simulation: A Survey
  5. 2026Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos
  6. 2026AgentGarten: Code Worlds for Evolving Agents
  7. 2026EmbodiedGen V2: An Agentic, Simulation-Ready 3D World Engine for Embodied AI
  8. 2026Exploratory, Communicative, and Deployable: Vision-Driven Embodied Agents for Open-World Mobile Manipulation