mlsanityData to 14 Oct 2026

What Do Latent Action Models Actually Learn?

arXiv 2506.15691 ↗First posted 27 May 2025cs.LG cs.AIcited by 32

Not on the map. Why?

Abstract (arXiv)

Latent action models (LAMs) aim to learn action-relevant changes from unlabeled videos by compressing changes between frames as latents. However, differences between video frames can be caused by controllable changes as well as exogenous noise, leading to an important concern -- do latents capture the changes caused by actions or irrelevant noise? This paper studies this issue analytically, presenting a linear model that encapsulates the essence of LAM learning, while being tractable.This provides several insights, including connections between LAM and principal component analysis (PCA), desiderata of the data-generating policy, and justification of strategies to encourage learning controllable changes using data augmentation, data cleaning, and auxiliary action-prediction. We also provide illustrative results based on numerical simulation, shedding light on the specific structure of observations, actions, and noise in data that influence LAM learning.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 28 of its 40 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

28 of 40 references.

References · 28

  1. 2017Neural Discrete Representation Learningcited by 4,712
  2. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  3. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
  4. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  5. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
  6. 2024Genie: Generative Interactive Environmentscited by 747
  7. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  8. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 405
  9. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
  10. 2024Latent Action Pretraining from Videoscited by 321
  11. 2023RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shotcited by 183
  12. 2023Structured World Models from Human Videoscited by 147
16 more
  1. 2023Learning to Act without Actionscited by 121
  2. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 114
  3. 2018Imitating Latent Policies from Observationcited by 101
  4. 2019Learning Action Representations for Reinforcement Learningcited by 75
  5. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AIcited by 72
  6. 2021Playable Video Generationcited by 64
  7. 2022Efficient Planning in a Compact Latent Action Spacecited by 57
  8. 2024DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Controlcited by 44
  9. 2025Latent Action Learning Requires Supervision in the Presence of Distractorscited by 44
  10. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  11. 2021LASER: Learning a Latent Action Space for Efficient Reinforcement Learningcited by 32
  12. 2018Learning what you can do before doing anythingcited by 26
  13. 2025Modeling Fine-Grained Hand-Object Dynamics for Egocentric Video Representation Learningcited by 20
  14. 2024AD3: Implicit Action is the Key for World Models to Distinguish the Diverse Visual Distractorscited by 10
  15. 2024Video Creation by Demonstrationcited by 8
  16. 2022Simple Emergent Action Representations from Multi-Task Policy Trainingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 27.

32 papers.

Sort:

By area:

  1. 2025Motus: A Unified Latent Action World Modelcited by 251
  2. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 135
  3. 2026VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Modelcited by 118
  4. 2025villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Modelscited by 63
  5. 2025True Self-Supervised Novel View Synthesis is Transferablecited by 15
  6. 2026MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstructioncited by 14
  7. 2026LARY: A Latent Action Representation Yielding Benchmark for Generalizable Vision-to-Action Alignmentcited by 12
  8. 2025LatBot: Distilling Universal Latent Actions for Vision-Language-Action Modelscited by 10
  9. 2026World2Act: Latent Action Post-Training from World Model Dynamicscited by 8
  10. 2026Learning Additively Compositional Latent Actions for Embodied AIcited by 7
  11. 2026Segment to Focus: Guiding Latent Action Models in the Presence of Distractorscited by 6
  12. 2026From Pixels to Tokens: A Systematic Study of Latent Action Supervision for Vision-Language-Action Modelscited by 4
20 more
  1. 2026Learning Visual Feature-Based World Models via Residual Latent Actioncited by 4
  2. 2026Vision-Language Models Unlock Task-Centric Latent Actionscited by 3
  3. 2026Grounding Generated Videos in Feasible Plans via World Modelscited by 3
  4. 2026ALAM: Algebraically Consistent Latent Action Model for Vision-Language-Action Modelscited by 3
  5. 2026Causally Debiased Latent Action Model for Embodied Action Conditioned World Modelscited by 3
  6. 2026From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulationcited by 2
  7. 2026Why Latent Actions Fail, and How to Prevent Itcited by 2
  8. 2025VITA: Zero-Shot Value Functions via Test-Time Adaptation of Vision-Language Modelscited by 1
  9. 2026SCAR: Self-Supervised Continuous Action Representation Learningcited by 1
  10. 2026FlexLAM: Resolving the Bottleneck Trade-off in Latent Action Learningcited by 1
  11. 2026DLAM: Distributional Latent Actions with Temporal Constraintscited by 1
  12. 2026Improving Cross-embodiment Transfer in Latent Action Models with Action-Similarity Supervisioncited by 1
  13. 2026Render, Don't Decode: Weight-Space World Models with Latent Structural Disentanglementcited by 0
  14. 2026Motion-Focused Latent Action Enables Cross-Embodiment VLA Training from Human EgoVideoscited by 0
  15. 2026Latent-Action-Guided Video-Language Feature Learning for Surgical Instrument-Tissue Interaction Recognitioncited by 0
  16. 2026Reconstructing Is Not Acting: Action-Centric Latent Dynamics Modelingcited by 0
  17. 2026Algebraic Consistency Alone Does Not Certify Temporal Structure in Latent Action Modelscited by 0
  18. 2026Devol-ONE: One Autoregressive Mixture of Transformers to Unify Vision-Language-Action and Latent World Modelingcited by 0
  19. 2026OneLA: Learning Many Worlds from Latent Actionscited by 0
  20. 2026VITAL: From Visual Motion to Precise Actioncited by 0