mlsanityData to 14 Oct 2026

GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Control

arXiv 2412.11198 ↗First posted 15 Dec 2024cs.CVcited by 78

On the map: World models → Embodied and robot world models

Abstract (arXiv)

We present GEM, a Generalizable Ego-vision Multimodal world model that predicts future frames using a reference frame, sparse features, human poses, and ego-trajectories. Hence, our model has precise control over object dynamics, ego-agent motion and human poses. GEM generates paired RGB and depth outputs for richer spatial understanding. We introduce autoregressive noise schedules to enable stable long-horizon generations. Our dataset is comprised of 4000+ hours of multimodal data across domains like autonomous driving, egocentric human activities, and drone flights. Pseudo-labels are used to get depth maps, ego-trajectories, and human poses. We use a comprehensive evaluation framework, including a new Control of Object Manipulation (COM) metric, to assess controllability. Experiments show GEM excels at generating diverse, controllable scenarios and temporal consistency over long generations. Code, models, and datasets are fully open-sourced.

Key facts. Cited by 78 papers in the map (data through 14 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 72 of its 86 references that are in the map. First posted on arXiv 15 Dec 2024.

Builds on

72 of 86 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024LIMT: Language-Informed Multi-Task Visual World Models18 Jul

Other references · 71

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  9. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  10. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  11. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  12. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
38 more
  1. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  2. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  3. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  5. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  6. 2024Depth Anything V2cited by 1,422
  7. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,360
  8. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  9. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,214
  10. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  11. 2021DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Camerascited by 687
  12. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 681
  13. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  14. 2024Movie Gen: A Cast of Media Foundation Modelscited by 644
  15. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  16. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 579
  17. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  18. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  19. 2023Learning Interactive Real-World Simulatorscited by 458
  20. 2018Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Drivingcited by 457
  21. 2024YOLOv11: An Overview of the Key Architectural Enhancementscited by 425
  22. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  23. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 350
  24. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  25. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 319
  26. 2023Effective Whole-body Pose Estimation with Two-stages Distillationcited by 311
  27. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 261
  28. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 255
  29. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Predictioncited by 233
  30. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Drivingcited by 233
  31. 2024DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videoscited by 204
  32. 2024MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidancecited by 198
  33. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 191
  34. 2022A Self-Supervised Descriptor for Image Copy Detectioncited by 176
  35. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  36. 2016End-to-end Learning of Driving Models from Large-scale Video Datasetscited by 148
  37. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 143
  38. 2021DriveGAN: Towards a Controllable High-Quality Neural Simulationcited by 125

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 36; 2026: 42.

78 papers.

Sort:

By area:

  1. 2025GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Drivingcited by 185
  2. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 166
  3. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 114
  4. 2025Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Thinkcited by 102
  5. 2025Stable Video Infinity: Infinite-Length Video Generation with Error Recyclingcited by 73
  6. 2025A Comprehensive Survey on World Models for Embodied AIcited by 68
  7. 20253D and 4D World Modeling: A Surveycited by 67
  8. 2025A Survey of World Models for Autonomous Drivingcited by 60
  9. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 51
  10. 2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving Worldcited by 47
  11. 2025Boosting Generative Image Modeling via Joint Image-Feature Synthesiscited by 45
  12. 2025Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidancecited by 45
38 more
  1. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 44
  2. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 38
  3. 2026Do World Action Models Generalize Better than VLAs? A Robustness Studycited by 36
  4. 2025VaViM and VaVAM: Autonomous Driving through Video Generative Modelingcited by 35
  5. 2025OmniNWM: Omniscient Driving Navigation World Modelscited by 33
  6. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  7. 2026WorldMark: A Unified Benchmark Suite for Interactive Video World Modelscited by 26
  8. 20253D Scene Generation: A Surveycited by 24
  9. 2025The Reality Gap in Robotics: Challenges, Solutions, and Best Practicescited by 24
  10. 2025UniFuture: A 4D Driving World Model for Future Generation and Perceptioncited by 22
  11. 2026UniDrive-WM: Unified Understanding, Planning and Generation World Model for Autonomous Drivingcited by 20
  12. 2025StateSpaceDiffuser: Bringing Long Context to Diffusion World Modelscited by 19
  13. 2025GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Controlcited by 19
  14. 2025MobileWorldBench: Towards Semantic World Modeling For Mobile Agentscited by 18
  15. 2025Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Modelscited by 17
  16. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  17. 2026Bridging Scene Generation and Planning: Driving with World Model via Unifying Vision and Motion Representationcited by 13
  18. 2026Egocentric World Model for Photorealistic Hand-Object Interaction Synthesiscited by 12
  19. 2026ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Drivingcited by 12
  20. 2026Learning Vision-Language-Action World Models for Autonomous Drivingcited by 11
  21. 2025UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Drivingcited by 10
  22. 2026MindDriver: Introducing Progressive Multimodal Reasoning for Autonomous Drivingcited by 10
  23. 2025CoVAR: Co-generation of Video and Action for Robotic Manipulation via Multi-Modal Diffusioncited by 9
  24. 2025From Generation to Generalization: Emergent Few-Shot Learning in Video Diffusion Modelscited by 8
  25. 2025EgoControl: Controllable Egocentric Video Generation via 3D Full-Body Posescited by 8
  26. 2026MAD: Motion Appearance Decoupling for efficient Driving World Modelscited by 8
  27. 2026InstaDrive: Instance-Aware Driving World Models for Realistic and Consistent Video Generationcited by 8
  28. 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trendscited by 8
  29. 2026OccAny: Generalized Unconstrained Urban 3D Occupancycited by 6
  30. 2026Overcoming Statistical Bias in Action-Controllable World Modelscited by 6
  31. 2025EvolvingAgent: Curriculum Self-evolving Agent with Continual World Model for Long-Horizon Taskscited by 5
  32. 2025Other Vehicle Trajectories Are Also Needed: A Driving World Model Unifies Ego-Other Vehicle Trajectories in Video Latent Spacecited by 5
  33. 2026CausalDrive: Real-time Causal World Models for Autonomous Drivingcited by 5
  34. 2026Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Jointscited by 4
  35. 2026PhysAlign: Physics-Coherent Image-to-Video Generation through Feature and 3D Representation Alignmentcited by 4
  36. 2026CounterScene: Counterfactual Causal Reasoning in Generative World Models for Safety-Critical Closed-Loop Evaluationcited by 4
  37. 2025Seeing Clearly, Forgetting Deeply: Revisiting Fine-Tuned Video Generators for Driving Simulationcited by 3
  38. 2025LangDriveCTRL: Natural Language Controllable Driving Scene Editing with Multi-modal Agentscited by 3

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data
  2. 2026Walk through Paintings: Egocentric World Models from Internet Priors
  3. 2026Ego-Human Motion Prediction with 3D-Aware LLM
  4. 2026Beyond Visual Quality: Evaluating Physical Consistency under Ego-Motion with EgoGenEval
  5. 2026EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoning
  6. 2024EgoGen: An Egocentric Synthetic Data Generator
  7. 2026Multi-Agent Egocentric World Model with Fine-Grained Embodied Interaction
  8. 2026AnyWorld: Factorized Egocentric World Models for Cross-Embodiment Generalization