mlsanityData to 7 Oct 2026

Pandora: Towards General World Model with Natural Language Actions and Video States

arXiv 2406.09455 ↗First posted 12 Jun 2024cs.CV cs.AI cs.CLcited by 83

On the map: World models → Video-generation world models and simulators

Abstract (arXiv)

World models simulate future states of the world in response to different actions. They facilitate interactive content creation and provides a foundation for grounded, long-horizon reasoning. Current foundation models do not fully meet the capabilities of general world models: large language models (LLMs) are constrained by their reliance on language modality and their limited understanding of the physical world, while video models lack interactive action control over the world simulations. This paper makes a step towards building a general world model by introducing Pandora, a hybrid autoregressive-diffusion model that simulates world states by generating videos and allows real-time control with free-text actions. Pandora achieves domain generality, video consistency, and controllability through large-scale pretraining and instruction tuning. Crucially, Pandora bypasses the cost of training-from-scratch by integrating a pretrained LLM (7B) and a pretrained video model, requiring only additional lightweight finetuning. We illustrate extensive outputs by Pandora across diverse domains (indoor/outdoor, natural/urban, human/robot, 2D/3D, etc.). The results indicate great potential of building stronger general world models with larger-scale training.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 69 of its 81 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

69 of 81 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Genie: Generative Interactive Environments23 Feb
  2. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  3. 2023Learning Interactive Real-World Simulators9 Oct
  4. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  5. 2022Scalable Diffusion Models with Transformers19 Dec
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  7. 2020Denoising Diffusion Probabilistic Models19 Jun
  8. 2017Matterport3D: Learning from RGB-D Data in Indoor Environments18 Sep

Other references · 61

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  9. 2022Video Diffusion Modelscited by 2,280
  10. 2024Mixtral of Expertscited by 1,785
  11. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  12. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
38 more
  1. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  2. 2018World Modelscited by 1,444
  3. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  4. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  5. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  6. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  7. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  8. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  9. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  10. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  11. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  12. 2019Model-Based Reinforcement Learning for Ataricited by 495
  13. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  14. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  15. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  16. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  17. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 343
  18. 2023ControlVideo: Training-free Controllable Text-to-Video Generationcited by 324
  19. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 320
  20. 2022Transformers are Sample-Efficient World Modelscited by 317
  21. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 309
  22. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 307
  23. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  24. 2022Flexible Diffusion Modeling of Long Videoscited by 294
  25. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  26. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Drivingcited by 230
  27. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 220
  28. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  29. 2024RoboDreamer: Learning Compositional World Models for Robot Imaginationcited by 187
  30. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 186
  31. 2024Vidu: a Highly Consistent, Dynamic and Skilled Text-to-Video Generator with Diffusion Modelscited by 159
  32. 2022TransDreamer: Reinforcement Learning with Transformer World Modelscited by 143
  33. 2022The Debate Over Understanding in AI's Large Language Modelscited by 142
  34. 2023Transformer-based World Models Are Happy With 100k Interactionscited by 137
  35. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Modelscited by 123
  36. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 118
  37. 2023Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusioncited by 105
  38. 2018Learning to Navigate in Cities Without a Mapcited by 103

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 39; 2026: 24.

83 papers.

Sort:

By area:

  1. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  2. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  3. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  4. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  5. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 168
  6. 2024Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generationcited by 167
  7. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  8. 2025WorldScore: A Unified Evaluation Benchmark for World Generationcited by 143
  9. 2026GigaWorld-Policy: An Efficient Action-Centered World--Action Modelcited by 129
  10. 2024GameGen-X: Interactive Open-world Game Video Generationcited by 124
  11. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representationcited by 123
  12. 2025WorldModelBench: Judging Video Generation Models As World Modelscited by 116
38 more
  1. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 108
  2. 2024DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformerscited by 96
  3. 2025TesserAct: Learning 4D Embodied World Modelscited by 90
  4. 2024ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restorationcited by 89
  5. 2025WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generationcited by 87
  6. 2024The Matrix: Infinite-Horizon World Generation with Real-Time Moving Controlcited by 79
  7. 2024GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Controlcited by 76
  8. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  9. 2025Long-Context State-Space Video World Modelscited by 59
  10. 2025VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Priorcited by 49
  11. 2025Pre-Trained Video Generative Models as World Simulatorscited by 43
  12. 2024Autoregressive Models in Vision: A Surveycited by 39
  13. 2024AVID: Adapting Video Diffusion Models to World Modelscited by 35
  14. 2024Multi-Task Interactive Robot Fleet Learning with Visual World Modelscited by 33
  15. 2024Learning Action and Reasoning-Centric Image Editing from Videos and Simulationscited by 32
  16. 2024The Dawn of Video Generation: Preliminary Explorations with SORA-like Modelscited by 30
  17. 2024SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video Generationcited by 28
  18. 2024Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generationcited by 28
  19. 2025Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tilecited by 26
  20. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Controlcited by 26
  21. 2025Generative Physical AI in Vision: A Surveycited by 24
  22. 2025Critique of World Modelcited by 24
  23. 2025LoViC: Efficient Long Video Generation with Context Compressioncited by 19
  24. 2025Playing with Transformer at 30+ FPS via Next-Frame Diffusioncited by 18
  25. 2025NeuralOS: Towards Simulating Operating Systems via Neural Generative Modelscited by 17
  26. 2025StateSpaceDiffuser: Bringing Long Context to Diffusion World Modelscited by 16
  27. 2025A Step Toward World Models: A Survey on Robotic Manipulationcited by 15
  28. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  29. 2024MMTrail: A Multimodal Trailer Video Dataset with Language and Music Descriptionscited by 13
  30. 2026Robotic Video World Models: A Survey of Applications, Research Challenges, Future Directionscited by 12
  31. 2025DMWM: Dual-Mind World Model with Long-Term Imaginationcited by 10
  32. 2025SimWorld: An Open-ended Realistic Simulator for Autonomous Agents in Physical and Social Worldscited by 10
  33. 2026Action100M: A Large-scale Video Action Datasetcited by 10
  34. 2026MemoBench: Benchmarking World Modeling in Dynamically Changing Environmentscited by 10
  35. 2026GigaWorld-Policy-0.5: A Faster and Stronger WAM Empowered by AutoResearchcited by 10
  36. 2026Code World Model: Coding Agent as World Braincited by 9
  37. 2024Video Creation by Demonstrationcited by 8
  38. 2025AnimeShooter: A Multi-Shot Animation Dataset for Reference-Guided Video Generationcited by 8

50 most cited of 83 All 83 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024WorldGPT: Empowering LLM as Multimodal World Model
  2. 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends
  3. 2026World Tokens: Enhancing Embodied Policies with Training-Time World Modeling
  4. 2026Worldscape-MoE: A Unified Mixture-of-Experts World Model for Scalable Heterogeneous Action Control
  5. 2026World-Language-Action Model for Unified World Modeling, Language Reasoning, and Action Synthesis
  6. 2026OpenWAM: An Open, Modular Exploration Towards Systematic World-Action Model Pretraining
  7. 2024WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokens
  8. 2025PAN: A World Model for General, Actionable, and Long-Horizon World Simulation