mlsanityData to 7 Oct 2026

AdaWorld: Learning Adaptable World Models with Latent Actions

arXiv 2503.18938 ↗First posted 24 Mar 2025cs.AI cs.CV cs.LGcited by 108

On the map: World models → Embodied and robot world models

Abstract (arXiv)

World models aim to learn action-controlled future prediction and have proven essential for the development of intelligent agents. However, most existing world models rely heavily on substantial action-labeled data and costly training, making it challenging to adapt to novel environments with heterogeneous actions through limited interactions. This limitation can hinder their applicability across broader domains. To overcome this limitation, we propose AdaWorld, an innovative world model learning approach that enables efficient adaptation. The key idea is to incorporate action information during the pretraining of world models. This is achieved by extracting latent actions from videos in a self-supervised manner, capturing the most critical transitions between frames. We then develop an autoregressive world model that conditions on these latent actions. This learning paradigm enables highly adaptable world models, facilitating efficient transfer and learning of new actions even with limited interactions and finetuning. Our comprehensive experiments across multiple environments demonstrate that AdaWorld achieves superior performance in both simulation quality and visual planning.

Key facts. Cited by 108 papers in the map (data through 7 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 106 of its 130 references that are in the map. First posted on arXiv 24 Mar 2025.

Builds on

106 of 130 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Control15 Dec
  2. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AI17 Oct
  3. 2024GenRL: Multimodal-foundation world models for generalization in embodied agents26 Jun
  4. 2024IRASim: A Fine-Grained World Model for Robot Manipulation20 Jun
  5. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Models24 May
  6. 2024RoboDreamer: Learning Compositional World Models for Robot Imagination18 Apr
  7. 20243D-VLA: A 3D Vision-Language-Action Generative World Model14 Mar
  8. 2023Learning to Act without Actions17 Dec
  9. 2023Structured World Models from Human Videos21 Aug
  10. 2022Masked World Models for Visual Control28 Jun

Other references · 96

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2022Scalable Diffusion Models with Transformerscited by 5,931
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  8. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  9. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  10. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  11. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  12. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
38 more
  1. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  2. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  3. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  4. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  5. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  6. 2023Mastering Diverse Domains through World Modelscited by 1,193
  7. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  8. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  9. 2016Deep Variational Information Bottleneckcited by 915
  10. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  11. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  12. 2022A Generalist Agentcited by 802
  13. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  14. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  15. 2024Genie: Generative Interactive Environmentscited by 706
  16. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 634
  17. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  18. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  19. 2019Model-Based Reinforcement Learning for Ataricited by 495
  20. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  21. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  22. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  23. 2023Learning Interactive Real-World Simulatorscited by 440
  24. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 385
  25. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  26. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 357
  27. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  28. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  29. 2022Transformers are Sample-Efficient World Modelscited by 317
  30. 2024Latent Action Pretraining from Videoscited by 302
  31. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 285
  32. 2024Diffusion Models Are Real-Time Game Enginescited by 282
  33. 2022Unifying Flow, Stereo and Depth Estimationcited by 270
  34. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
  35. 2024Navigation World Modelscited by 262
  36. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  37. 2022Multi-Game Decision Transformerscited by 212
  38. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200

50 most cited of 96 All 96 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 29; 2026: 79.

108 papers.

Sort:

By area:

  1. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  2. 2025Motus: A Unified Latent Action World Modelcited by 231
  3. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 168
  4. 2025Ctrl-World: A Controllable Generative World Model for Robot Manipulationcited by 149
  5. 2025Unified Vision-Language-Action Modelcited by 128
  6. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 117
  7. 2026VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Modelcited by 104
  8. 2025Vid2World: Crafting Video Diffusion Models to Interactive World Modelscited by 67
  9. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  10. 20253D and 4D World Modeling: A Surveycited by 64
  11. 2026Motubrain: An Advanced World Action Model for Robot Controlcited by 54
  12. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
38 more
  1. 2026LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policiescited by 46
  2. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 42
  3. 2026Learning Latent Action World Models In The Wildcited by 42
  4. 2026RISE: Self-Improving Robot Policy with Compositional World Modelcited by 39
  5. 2026VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Modelcited by 39
  6. 2025CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learningcited by 37
  7. 2026Do World Action Models Generalize Better than VLAs? A Robustness Studycited by 35
  8. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 33
  9. 2026MIND: Benchmarking Memory Consistency and Action Control in World Modelscited by 31
  10. 2025What Do Latent Action Models Actually Learn?cited by 30
  11. 2026Native Video-Action Pretraining for Generalizable Robot Controlcited by 30
  12. 2026LaST0: Latent Spatio-Temporal Chain-of-Thought for Robotic Vision-Language-Action Modelcited by 28
  13. 2025ViPRA: Video Prediction for Robot Actionscited by 24
  14. 2025A Survey of Interactive Generative Videocited by 23
  15. 2025MLA: A Multisensory Language-Action Model for Multimodal Understanding and Forecasting in Robotic Manipulationcited by 22
  16. 2025Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Modelscited by 17
  17. 2026AdaJEPA: An Adaptive Latent World Modelcited by 16
  18. 2025A Step Toward World Models: A Survey on Robotic Manipulationcited by 15
  19. 2026WALL-WM: Carving World Action Modeling at the Event Jointscited by 14
  20. 2025True Self-Supervised Novel View Synthesis is Transferablecited by 13
  21. 2025ManualVLA: A Unified VLA Model for Chain-of-Thought Manual Generation and Robotic Manipulationcited by 13
  22. 2025Mitty: Diffusion-based Human-to-Robot Video Generationcited by 13
  23. 2025Co-Evolving Latent Action World Modelscited by 12
  24. 2026Robotic Video World Models: A Survey of Applications, Research Challenges, Future Directionscited by 12
  25. 2026WorldCam: Interactive Autoregressive 3D Gaming Worlds with Camera Pose as a Unifying Geometric Representationcited by 12
  26. 2026MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstructioncited by 11
  27. 2026VideoWorld 2: Learning Transferable Knowledge from Real-world Videoscited by 11
  28. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modelingcited by 11
  29. 2025StaMo: Unsupervised Learning of Generalizable Robot Motion from Compact State Representationcited by 10
  30. 2026LARY: A Latent Action Representation Yielding Benchmark for Generalizable Vision-to-Action Alignmentcited by 10
  31. 2026World Action Models: A Surveycited by 10
  32. 2026Code World Model: Coding Agent as World Braincited by 9
  33. 2026Olaf-World: Orienting Latent Actions for Video World Modelingcited by 8
  34. 2026Chain of World: World Model Thinking in Latent Motioncited by 8
  35. 2026VISTA: Triplet-Supervised Video Style Transfer with Diffusion Transformerscited by 8
  36. 2026World2Act: Latent Action Post-Training from World Model Dynamicscited by 7
  37. 2025Benchmarking Vision, Language, & Action Models in Procedurally Generated, Open Ended Action Environmentscited by 6
  38. 2026Learning to Act Robustly with View-Invariant Latent Actionscited by 6

50 most cited of 108 All 108 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning Transferable Dynamics Priors from Action to World Modeling
  2. 2026WorldAgen: Unified State-Action Prediction with Test-Time World Model Training
  3. 2026MiniWorld: Democratizing the Training of Video World Models from Scratch
  4. 2026Foresight Without Seeing: Latent Futures for World Action Models
  5. 2026World Tokens: Enhancing Embodied Policies with Training-Time World Modeling
  6. 2026X-Foresight: A Joint Vision-Action Causal Forecasting Network via Predictive World Modeling
  7. 2026DeltaWAM: Distilling Action Tangent Fields into World Action Models
  8. 2026World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetry