mlsanityData to 14 Oct 2026

Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control

arXiv 2503.14492 ↗First posted 18 Mar 2025cs.CV cs.AI cs.LGcited by 106

Not on the map. Why?

Cited by 39 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce Cosmos-Transfer, a conditional world generation model that can generate world simulations based on multiple spatial control inputs of various modalities such as segmentation, depth, and edge. In the design, the spatial conditional scheme is adaptive and customizable. It allows weighting different conditional inputs differently at different spatial locations. This enables highly controllable world generation and finds use in various world-to-world transfer use cases, including Sim2Real. We conduct extensive evaluations to analyze the proposed model and demonstrate its applications for Physical AI, including robotics Sim2Real and autonomous vehicle data enrichment. We further demonstrate an inference scaling strategy to achieve real-time world generation with an NVIDIA GB200 NVL72 rack. To help accelerate research development in the field, we open-source our models and code at https://github.com/nvidia-cosmos/cosmos-transfer1.

Key facts. Cited by 106 papers in the map (data through 14 Oct 2026). Builds on 59 of its 68 references that are in the map. First posted on arXiv 18 Mar 2025.

Builds on

59 of 68 references.

References · 59

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  3. 2022Scalable Diffusion Models with Transformerscited by 6,054
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  6. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  7. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  8. 2024Depth Anything V2cited by 1,422
  9. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,351
  10. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
  11. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,274
  12. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,179
38 more
  1. 2021Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Datacited by 970
  2. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  3. 2017Unsupervised Learning of Depth and Ego-Motion from Videocited by 782
  4. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 781
  5. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  6. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
  7. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  8. 2023Universal Guidance for Diffusion Modelscited by 437
  9. 2018Video-to-Video Synthesiscited by 390
  10. 2023Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Modelscited by 359
  11. 2025GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Controlcited by 341
  12. 2022Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectivescited by 320
  13. 2023Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detectioncited by 315
  14. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 313
  15. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 304
  16. 2024Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillationcited by 249
  17. 2023UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wildcited by 200
  18. 2019Few-shot Video-to-Video Synthesiscited by 197
  19. 2024Pixtral 12Bcited by 179
  20. 2018PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planningcited by 150
  21. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  22. 2024PIXART-δ: Fast and Controllable Image Generation with Latent Consistency Modelscited by 141
  23. 2022Semantic Image Synthesis via Diffusion Modelscited by 132
  24. 2020You Only Need Adversarial Supervision for Semantic Image Synthesiscited by 129
  25. 2023HumanSD: A Native Skeleton-Guided Diffusion Model for Human Image Generationcited by 105
  26. 2020RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Realcited by 91
  27. 2023HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splattingcited by 89
  28. 2023PEEKABOO: Interactive Video Generation via Masked-Diffusioncited by 76
  29. 2021Multimodal Conditional Image Synthesis with Product-of-Experts GANscited by 75
  30. 2020RetinaGAN: An Object-aware Approach to Sim-to-Real Transfercited by 70
  31. 2020World-Consistent Video-to-Video Synthesiscited by 69
  32. 2023HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusioncited by 68
  33. 2023Freestyle Layout-to-Image Synthesiscited by 67
  34. 2024Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Modelcited by 66
  35. 2021Robot Learning from Randomized Simulations: A Reviewcited by 63
  36. 2024InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Modelscited by 56
  37. 2023Scenario Diffusion: Controllable Driving Scenario Generation With Diffusioncited by 47
  38. 2021Image Synthesis via Semantic Compositioncited by 45

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 34; 2026: 72.

106 papers.

Sort:

By area:

  1. 2025World Simulation with Video Foundation Models for Physical AIcited by 200
  2. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 179
  3. 2025ViPE: Video Pose Engine for 3D Geometric Perceptioncited by 170
  4. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 165
  5. 2025HunyuanWorld 1.0: Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixelscited by 94
  6. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 82
  7. 2025A Comprehensive Survey on World Models for Embodied AIcited by 68
  8. 20253D and 4D World Modeling: A Surveycited by 67
  9. 2025GigaWorld-0: World Models as Data Engine to Empower Embodied AIcited by 62
  10. 2026World Action Models: The Next Frontier in Embodied AIcited by 50
  11. 2025Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systemscited by 48
  12. 2025GigaBrain-0: A World Model-Powered Vision-Language-Action Modelcited by 44
38 more
  1. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 38
  2. 2025SimScale: Learning to Drive via Real-World Simulation at Scalecited by 36
  3. 2025PAI-Bench: A Comprehensive Benchmark For Physical AIcited by 34
  4. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Controlcited by 28
  5. 2025Do What You Say: Steering Vision-Language-Action Models via Runtime Reasoning-Action Alignment Verificationcited by 26
  6. 2026GigaBrain-0.5M*: a VLA That Learns From World Model-Based Reinforcement Learningcited by 24
  7. 2025PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generationcited by 23
  8. 2026RealWonder: Real-Time Physical Action-Conditioned Video Generationcited by 22
  9. 2025ORV: 4D Occupancy-centric Robot Video Generationcited by 17
  10. 2026Fail2Drive: Benchmarking Closed-Loop Driving Generalizationcited by 17
  11. 2025Think Before You Diffuse: Infusing Physical Rules into Video Diffusioncited by 16
  12. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  13. 2026Physical Simulator In-the-Loop Video Generationcited by 14
  14. 2026iWorld-Bench: A Benchmark for Interactive World Models with a Unified Action Generation Frameworkcited by 14
  15. 2026Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoningcited by 14
  16. 2025WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Drivingcited by 13
  17. 2025EMMA: Generalizing Real-World Robot Manipulation via Generative Visual Transfercited by 12
  18. 2026World Action Models: A Surveycited by 12
  19. 2025TC-Light: Temporally Coherent Generative Rendering for Realistic World Transfercited by 11
  20. 2025AnchorDream: Repurposing Video Diffusion for Embodiment-Aware Robot Data Synthesiscited by 11
  21. 2026GigaWorld-Policy-0.5: A Faster and Stronger WAM Empowered by AutoResearchcited by 11
  22. 2025Towards foundational LiDAR world models with efficient latent flow matchingcited by 10
  23. 2025LongVie 2: Multimodal Controllable Ultra-Long Video World Modelcited by 9
  24. 2026UniDriveDreamer: A Single-Stage Multimodal World Model for Autonomous Drivingcited by 9
  25. 2026ReWorld: Multi-Dimensional Reward Modeling for Embodied World Modelscited by 8
  26. 2026RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learningcited by 8
  27. 2025RoboTransfer: Controllable Geometry-Consistent Video Diffusion for Manipulation Policy Transfercited by 7
  28. 2026RoboVIP: Multi-View Video Generation with Visual Identity Prompting Augments Robot Manipulationcited by 7
  29. 2026InSpatio-WorldFM: An Open-Source Real-Time Generative Frame Modelcited by 7
  30. 2025CtrlVDiff: Controllable Video Generation via Unified Multimodal Video Diffusioncited by 6
  31. 2026World-Action Models for Robot Learning and Control: A Surveycited by 6
  32. 2025Simulating the Unseen: Crash Prediction Must Learn from What Did Not Happencited by 5
  33. 2025TC-LoRA: Temporally Modulated Conditional LoRA for Adaptive Diffusion Controlcited by 5
  34. 2025NeuralRemaster: Phase-Preserving Diffusion for Structure-Aligned Generationcited by 5
  35. 2026Driving with DINO: Vision Foundation Features as a Unified Bridge for Sim-to-Real Generation in Autonomous Drivingcited by 5
  36. 2026EditCtrl: Disentangled Local and Global Control for Real-Time Generative Video Editingcited by 5
  37. 2026Worldscape-MoE: A Unified Mixture-of-Experts World Model for Scalable Heterogeneous Action Controlcited by 5
  38. 2025VHOI: Controllable Video Generation of Human-Object Interactions from Sparse Trajectories via Motion Densificationcited by 4

50 most cited of 106 All 106 (JSON)