mlsanityData to 13 Oct 2026

Learning to Act without Actions

arXiv 2312.10812 ↗First posted 17 Dec 2023cs.LG cs.AIcited by 120

On the map: World models → Embodied and robot world models

Abstract (arXiv)

Pre-training large models on vast amounts of web data has proven to be an effective approach for obtaining powerful, general models in domains such as language and vision. However, this paradigm has not yet taken hold in reinforcement learning. This is because videos, the most abundant form of embodied behavioral data on the web, lack the action labels required by existing methods for imitating behavior from demonstrations. We introduce Latent Action Policies (LAPO), a method for recovering latent action information, and thereby latent-action policies, world models, and inverse dynamics models, purely from videos. LAPO is the first method able to recover the structure of the true action space just from observed dynamics, even in challenging procedurally-generated environments. LAPO enables training latent-action policies that can be rapidly fine-tuned into expert-level policies, either offline using a small action-labeled dataset, or online with rewards. LAPO takes a first step towards pre-training powerful, generalist policies and world models on the vast amounts of videos readily available on the web.

Key facts. Cited by 120 papers in the map (data through 13 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 34 of its 49 references that are in the map. First posted on arXiv 17 Dec 2023.

Builds on

34 of 49 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,106
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  4. 2020Language Models are Few-Shot Learnerscited by 32,320
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,656
  6. 2017Proximal Policy Optimization Algorithmscited by 14,592
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,915
  8. 2017Neural Discrete Representation Learningcited by 4,680
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,284
  10. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,870
  11. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,056
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,889
22 more
  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,782
  2. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 871
  3. 2022A Generalist Agentcited by 803
  4. 2021SoundStream: An End-to-End Neural Audio Codeccited by 762
  5. 2020Jukebox: A Generative Model for Musiccited by 537
  6. 2018Behavioral Cloning from Observationcited by 483
  7. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 479
  8. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  9. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 339
  10. 2022Transformers are Sample-Efficient World Modelscited by 324
  11. 2018Generative Adversarial Imitation from Observationcited by 183
  12. 2016Variational Lossy Autoencodercited by 176
  13. 2018Imitating Latent Policies from Observationcited by 101
  14. 2018Playing hard exploration games by watching YouTubecited by 98
  15. 2019Recent Advances in Imitation Learning from Observationcited by 92
  16. 2023Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networkscited by 81
  17. 2020Reinforcement Learning with Videos: Combining Offline Observations with Interactioncited by 80
  18. 2023Reinforcement Learning from Passive Data via Latent Intentionscited by 65
  19. 2021Playable Video Generationcited by 64
  20. 2019Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreementcited by 46
  21. 2022Learning to Drive by Watching YouTube Videos: Action-Conditioned Contrastive Policy Pretrainingcited by 37
  22. 2022Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectoriescited by 27

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 1; 2024: 13; 2025: 43; 2026: 63.

120 papers.

Sort:

By area:

  1. 2024Genie: Generative Interactive Environmentscited by 721
  2. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 397
  3. 2024Latent Action Pretraining from Videoscited by 308
  4. 2025Motus: A Unified Latent Action World Modelcited by 239
  5. 2023Any-point Trajectory Modeling for Policy Learningcited by 210
  6. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 172
  7. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 153
  8. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 125
  9. 2026VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Modelcited by 112
  10. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 110
  11. 2025FLARE: Robot Learning with Implicit World Modelingcited by 108
  12. 2025Universal Actions for Enhanced Embodied Foundation Modelscited by 80
38 more
  1. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AIcited by 69
  2. 2024Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videoscited by 66
  3. 2025villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Modelscited by 59
  4. 2026LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policiescited by 49
  5. 2024DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Controlcited by 44
  6. 2025VideoWorld: Exploring Knowledge Learning from Unlabeled Videoscited by 42
  7. 2025Latent Action Learning Requires Supervision in the Presence of Distractorscited by 42
  8. 2025CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrationscited by 39
  9. 2025CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learningcited by 39
  10. 2024AVID: Adapting Video Diffusion Models to World Modelscited by 37
  11. 2025VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Modelscited by 34
  12. 2026World Guidance: World Modeling in Condition Space for Action Generationcited by 33
  13. 2026Native Video-Action Pretraining for Generalizable Robot Controlcited by 32
  14. 2025UniSkill: Imitating Human Videos via Cross-Embodiment Skill Representationscited by 30
  15. 2025What Do Latent Action Models Actually Learn?cited by 30
  16. 2025ViPRA: Video Prediction for Robot Actionscited by 24
  17. 2025Latent Action Pretraining Through World Modelingcited by 22
  18. 2026Sensorimotor World Models: Perception for Action via Inverse Dynamicscited by 19
  19. 2025AMPLIFY: Actionless Motion Priors for Robot Learning from Videoscited by 18
  20. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  21. 2026Next Forcing: Causal World Modeling with Multi-Chunk Predictioncited by 17
  22. 2025PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planningcited by 14
  23. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modelingcited by 14
  24. 2025Co-Evolving Latent Action World Modelscited by 13
  25. 2024Diffusion Imitation from Observationcited by 12
  26. 2026MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstructioncited by 12
  27. 2026VideoWorld 2: Learning Transferable Knowledge from Real-world Videoscited by 12
  28. 2025StaMo: Unsupervised Learning of Generalizable Robot Motion from Compact State Representationcited by 11
  29. 2026DynVLA: Learning World Dynamics for Action Reasoning in Autonomous Drivingcited by 11
  30. 2026RepWAM: World Action Modeling with Representation Visual-Action Tokenizerscited by 11
  31. 2024AD3: Implicit Action is the Key for World Models to Distinguish the Diverse Visual Distractorscited by 10
  32. 2026Scaling Behavior Cloning Improves Causal Reasoning: An Open Model for Real-Time Video Game Playingcited by 10
  33. 2026DeltaVLA: Prior-Guided Vision-Language-Action Models via World Knowledge Variationcited by 10
  34. 2026LARY: A Latent Action Representation Yielding Benchmark for Generalizable Vision-to-Action Alignmentcited by 10
  35. 2025Object-Centric Latent Action Learningcited by 9
  36. 2026Olaf-World: Orienting Latent Actions for Video World Modelingcited by 9
  37. 2024Pre-trained Visual Dynamics Representations for Efficient Policy Learningcited by 8
  38. 2025Latent Policy Steering with Embodiment-Agnostic Pretrained World Modelscited by 8

50 most cited of 120 All 120 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Latent Action World Models for Control with Unlabeled Trajectories
  2. 2026Learning Latent Action World Models In The Wild
  3. 2026Imitation from Heterogeneous Demonstrations using Grounded Latent-Action World Models
  4. 2026LAFP: Preserving Latent Action Structure in Latent Policy Learning via Flow Matching
  5. 2024A Dual Approach to Imitation Learning from Observations with Offline Datasets
  6. 2026CLAW: Learning Continuous Latent Action World Models via Adversarial Latent Regularization
  7. 2022Reinforcement Learning with Action-Free Pre-Training from Videos
  8. 2026LD4WAM: Learning Latent Dynamics from Human Videos for World Action Models