mlsanityData to 7 Oct 2026

Cosmos 3: Omnimodal World Models for Physical AI

arXiv 2606.02800 ↗First posted 1 Jun 2026cs.CV cs.AI cs.LGcited by 137

On the map: World models → Video-generation world models and simulators

Abstract (arXiv)

We introduce Cosmos 3, a family of omnimodal world models designed to jointly process and generate language, image, video, audio, and action sequences within a unified mixture-of-transformers architecture. By supporting highly flexible input-output configurations, Cosmos 3 seamlessly unifies critical modalities for Physical AI -- effectively subsuming vision-language models, video generators, world simulators, and world-action models into a single framework. Our evaluation demonstrates that Cosmos 3 establishes a new state-of-the-art across a diverse suite of understanding and generation tasks, demonstrating omnimodal world models as scalable, general-purpose backbones for embodied agents. Our post-trained Cosmos 3 models were ranked as the best open-source Text-to-Image and Image-to-Video models by Artificial Analysis, and the best policy model by RoboArena at the time the technical report was written. To accelerate open research and deployment in Physical AI, we make our code, model checkpoints, curated synthetic datasets, and evaluation benchmark available under the Linux Foundation's OpenMDW-1.1 License at https://github.com/nvidia/cosmos and https://huggingface.co/collections/nvidia/cosmos3. The project website is available at https://research.nvidia.com/labs/cosmos-lab/cosmos3.

Key facts. Cited by 137 papers in the map (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 136 of its 358 references that are in the map. First posted on arXiv 1 Jun 2026.

Builds on

136 of 358 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026Advancing Open-source World Models28 Jan
  2. 2025Wan: Open and Advanced Large-Scale Video Generative Models26 Mar
  3. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  4. 2023Learning Interactive Real-World Simulators9 Oct

Other references · 132

  1. 2025Qwen2.5-VL Technical Reportcited by 5,418
  2. 2024GPT-4o System Cardcited by 4,467
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  4. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  5. 2025Qwen3-VL Technical Reportcited by 2,534
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  7. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  8. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  9. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  10. 2018World Modelscited by 1,444
  11. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  12. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
38 more
  1. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  2. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  3. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  4. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  5. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  6. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  7. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
  8. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  9. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  10. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  11. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  12. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
  13. 2025Depth Anything 3: Recovering the Visual Space from Any Viewscited by 626
  14. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  15. 2024Movie Gen: A Cast of Media Foundation Modelscited by 614
  16. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  17. 2024LTX-Video: Realtime Video Latent Diffusioncited by 549
  18. 2025Qwen3-Omni Technical Reportcited by 451
  19. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  20. 2026World Action Models are Zero-shot Policiescited by 411
  21. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  22. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  23. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  24. 2026Causal World Modeling for Robot Controlcited by 289
  25. 2025Seedance 1.0: Exploring the Boundaries of Video Generation Modelscited by 247
  26. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 242
  27. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contextscited by 240
  28. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  29. 2026LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixelscited by 237
  30. 2025Motus: A Unified Latent Action World Modelcited by 231
  31. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  32. 2026LTX-2: Efficient Joint Audio-Visual Foundation Modelcited by 213
  33. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 211
  34. 2025Video models are zero-shot learners and reasonerscited by 208
  35. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 204
  36. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 198
  37. 2025MolmoAct: Action Reasoning Models that can Reason in Spacecited by 189
  38. 2024RoboDreamer: Learning Compositional World Models for Robot Imaginationcited by 187

50 most cited of 132 All 132 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 137.

137 papers.

Sort:

By area:

  1. 2026ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?cited by 33
  2. 2026From Foundation to Application: Improving VLA Models in Practicecited by 25
  3. 2026Xiaomi-Robotics-1: Scaling Vision-Language-Action Models with over 100K Hours of Real-World Trajectoriescited by 24
  4. 2026Causal-rCM: A Unified Teacher-Forcing and Self-Forcing Open Recipe for Autoregressive Diffusion Distillation in Streaming Video Generation and Interactive World Modelscited by 23
  5. 2026ActWorld: From Explorable to Interactive World Model via Action-Aware Memorycited by 13
  6. 2026Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligencecited by 13
  7. 2026Addressable Memory for Video World Modelscited by 11
  8. 2026GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluationcited by 10
  9. 2026GigaWorld-Policy-0.5: A Faster and Stronger WAM Empowered by AutoResearchcited by 10
  10. 2026EchoWM: Open and Enterable Omnimodal World Modelscited by 10
  11. 2026Flex-π: A Multi-Stream World-Action Model with Compute Flexibilitycited by 9
  12. 2026Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Drivingcited by 8
38 more
  1. 2026PhysisForcing: Physics Reinforced World Simulator for Robotic Manipulationcited by 7
  2. 2026Physics-IQ Verifiedcited by 6
  3. 2026Sol Video Inference Engine: Agent-Native Full-Stack Acceleration Framework for Efficient Video Generationcited by 6
  4. 2026Orca: The World is in Your Mindcited by 6
  5. 2026WAM-TTT: Steering World-Action Models by Watching Human Play at Test Timecited by 6
  6. 2026ACE-Brain-0.5: A Unified Embodied Foundational Model for Physical Agentic AIcited by 5
  7. 2026RynnValue: Scaling Robotic Value Foundation Models with Temporal Distancecited by 5
  8. 2026Marionette: Predicting World States, Rendering Geometry, Painting Appearancecited by 5
  9. 2026PAIWorld: A 3D-Consistent World Foundation Model for Robotic Manipulationcited by 4
  10. 2026SC3-Eval: Evaluating Robot Foundation Models via Self-Consistent Video Generationcited by 4
  11. 2026Data Pyramid for Embodied Manipulation: A Surveycited by 4
  12. 2026A Definition and Roadmap for World Modelscited by 3
  13. 2026Xiaomi-Robotics-U0: Unified Embodied Synthesis with World Foundation Modelcited by 3
  14. 2026SceneActBench: Can Agents Act on the 3D Scenes They See?cited by 3
  15. 2026Enfold: Folding World Model Imagination into Predictive Representations for Ultra-Efficient Embodied Controlcited by 3
  16. 2026DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulationcited by 3
  17. 2026Long-Horizon Audio-Visual Generation for Persistent Stories and Interactive Worldscited by 3
  18. 2026SenseNova-U1.5: Towards Native Unified Visual Intelligencecited by 3
  19. 2026Critique of Agent Modelcited by 2
  20. 2026Show, Don't Tell: Evaluating Spatial Cognition in Generative Pixels Rather Than LLM Textcited by 2
  21. 2026SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generationcited by 2
  22. 2026Security of World-Model-Based Embodied AI: A Lifecycle of Threats, Defenses, and Evaluationcited by 2
  23. 2026EgoGenesis: Egocentric World-Action Modeling with Online Anchored Projective Memory and Action-3D RoPEcited by 2
  24. 2026PhiZero: A World Model Built Around Physical Languagecited by 2
  25. 2026Uncertainty-Aware World Model for Aerial Image-Goal Navigationcited by 2
  26. 2026WorldSimProbe: Diagnosing Simulator Faithfulness in Action-Conditioned World Models for Embodied Manipulationcited by 2
  27. 2026ForgeWM: Progressive Causal Training for Few-Step Action-Conditioned Video World Modelscited by 2
  28. 2026What Matters for Latent Actions in Robot Learningcited by 2
  29. 2026GeoWAM: Visual Geometry World Action Models for Autonomous Drivingcited by 2
  30. 2026Modality-Autoregressive World-Action Modelscited by 2
  31. 2026Learning Action Priors for Cross-embodiment Robot Manipulationcited by 1
  32. 2026RoboGaze: Evaluating Robot World Models via Structured Vision-Language Analysiscited by 1
  33. 2026RxBrain: Embodied Cognition Foundation Model with Joint Language-Visual Reasoning and Imaginationcited by 1
  34. 2026PhyAI: Real-Time Physical AI at the Edge, Scalable Rollouts in the Cloudcited by 1
  35. 2026HelloWorld: Enabling Socially Interactive Characters in Video World Modelscited by 1
  36. 2026CosmosAlign: Adapting a World Foundation Model for Generative Traffic Video Forecastingcited by 1
  37. 2026From Detection to Understanding: TAR and TAR-Bench for Multi-Task Traffic Anomaly Reasoningcited by 1
  38. 2026WorldRover: A Scalable Synthetic Video Data Engine for World Exploration with Rich Annotationscited by 1

50 most cited of 137 All 137 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Comprehensive Survey on World Models for Embodied AI
  2. 2025Adapting Vision-Language Models for Evaluating World Models
  3. 2025WorldLens: Full-Spectrum Evaluations of Driving World Models in Real World
  4. 2026Hy-Embodied-VLM-1.0: Efficient Physical-World Agents
  5. 20243D-VLA: A 3D Vision-Language-Action Generative World Model
  6. 2025WoW: Towards a World omniscient World model Through Embodied Interaction
  7. 2025EWMBench: Evaluating Scene, Motion, and Semantic Quality in Embodied World Models
  8. 2025Simulating the Real World: A Unified Survey of Multimodal Generative Models