mlsanityData to 14 Oct 2026

Latent Action Learning Requires Supervision in the Presence of Distractors

arXiv 2502.00379 ↗First posted 1 Feb 2025cs.CV cs.AI cs.LGcited by 44

Not on the map. Why?

Abstract (arXiv)

Recently, latent action learning, pioneered by Latent Action Policies (LAPO), have shown remarkable pre-training efficiency on observation-only data, offering potential for leveraging vast amounts of video available on the web for embodied AI. However, prior work has focused on distractor-free data, where changes between observations are primarily explained by ground-truth actions. Unfortunately, real-world videos contain action-correlated distractors that may hinder latent action learning. Using Distracting Control Suite (DCS) we empirically investigate the effect of distractors on latent action learning and demonstrate that LAPO struggle in such scenario. We propose LAOM, a simple LAPO modification that improves the quality of latent actions by 8x, as measured by linear probing. Importantly, we show that providing supervision with ground-truth actions, as few as 2.5% of the full dataset, during latent action learning improves downstream performance by 4.2x on average. Our findings suggest that integrating supervision during Latent Action Models (LAM) training is critical in the presence of distractors, challenging the conventional pipeline of first learning LAM and only then decoding from latent to ground-truth actions.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Builds on 61 of its 69 references that are in the map. First posted on arXiv 1 Feb 2025.

Builds on

61 of 69 references.

References · 61

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2017Neural Discrete Representation Learningcited by 4,712
  6. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  9. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  10. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  11. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
  12. 2024Genie: Generative Interactive Environmentscited by 747
38 more
  1. 2018Behavioral Cloning from Observationcited by 492
  2. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
  3. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  4. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  5. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 351
  6. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  7. 2024Latent Action Pretraining from Videoscited by 321
  8. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  9. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 137
  10. 2023Learning to Act without Actionscited by 121
  11. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 105
  12. 2018Imitating Latent Policies from Observationcited by 101
  13. 2021The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixelscited by 100
  14. 2018Playing hard exploration games by watching YouTubecited by 99
  15. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 95
  16. 2019Recent Advances in Imitation Learning from Observationcited by 93
  17. 2020Reinforcement Learning with Videos: Combining Offline Observations with Interactioncited by 82
  18. 2021DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representationscited by 74
  19. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AIcited by 72
  20. 2022Denoised MDPs: Learning World Models Better Than the World Itselfcited by 68
  21. 2023Reinforcement Learning from Passive Data via Latent Intentionscited by 68
  22. 2020Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstructioncited by 66
  23. 2021Learning Task Informed Abstractionscited by 58
  24. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 54
  25. 2022Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Modelscited by 45
  26. 2024DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Controlcited by 44
  27. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  28. 2022Learning to Drive by Watching YouTube Videos: Action-Conditioned Contrastive Policy Pretrainingcited by 37
  29. 2022Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learningcited by 34
  30. 2022Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectoriescited by 28
  31. 2022MineRL Diamond 2021 Competition: Overview, Results, and Lessons Learnedcited by 26
  32. 2022A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learningcited by 25
  33. 2021Learning Representations for Pixel-based Control: What Matters and Why?cited by 24
  34. 2023Learning World Models with Identifiable Factorizationcited by 23
  35. 2023RePo: Resilient Model-Based Reinforcement Learning by Regularizing Posterior Predictabilitycited by 18
  36. 2023Simplified Temporal Consistency Reinforcement Learningcited by 17
  37. 2022Dungeons and Data: A Large-Scale NetHack Datasetcited by 16
  38. 2022Light-weight probing of unsupervised representations for Reinforcement Learningcited by 15

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 37.

44 papers.

Sort:

By area:

  1. 2025Motus: A Unified Latent Action World Modelcited by 251
  2. 2026VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Modelcited by 118
  3. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 114
  4. 2025villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Modelscited by 63
  5. 2026Learning Latent Action World Models In The Wildcited by 47
  6. 2025CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrationscited by 41
  7. 2025CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learningcited by 40
  8. 2025What Do Latent Action Models Actually Learn?cited by 32
  9. 2026MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstructioncited by 14
  10. 2026DynVLA: Learning World Dynamics for Action Reasoning in Autonomous Drivingcited by 11
  11. 2026Olaf-World: Orienting Latent Actions for Video World Modelingcited by 10
  12. 2026DiLA: Disentangled Latent Action World Modelscited by 10
32 more
  1. 2025Object-Centric Latent Action Learningcited by 9
  2. 2026Learning to Act Robustly with View-Invariant Latent Actionscited by 8
  3. 2026Open-AoE: An Open Egocentric Manipulation Dataset and Toolchain for Embodied Learningcited by 8
  4. 2026Factored Latent Action World Modelscited by 7
  5. 2026Learning Additively Compositional Latent Actions for Embodied AIcited by 7
  6. 2026Segment to Focus: Guiding Latent Action Models in the Presence of Distractorscited by 6
  7. 2026Hierarchical Latent Action Modelcited by 6
  8. 2026LARA: Latent Action Representation Alignment for Vision-Language-Action Modelscited by 5
  9. 2026From Pixels to Tokens: A Systematic Study of Latent Action Supervision for Vision-Language-Action Modelscited by 4
  10. 2026What Matters for Latent Actions in Robot Learningcited by 4
  11. 2026Vision-Language Models Unlock Task-Centric Latent Actionscited by 3
  12. 2026Causally Debiased Latent Action Model for Embodied Action Conditioned World Modelscited by 3
  13. 2026JoyAI-RA 0.5: Scaling Robot Manipulation Learning via Dual Action Alignmentcited by 3
  14. 2026CoLA-Flow Policy: Temporally Coherent Imitation Learning via Continuous Latent Action Flow Matching for Robotic Manipulationcited by 2
  15. 2026From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulationcited by 2
  16. 2026Why Latent Actions Fail, and How to Prevent Itcited by 2
  17. 2026Dueling World Models: Advantage-Style Action Channels for Common-Mode Distractor Rejectioncited by 2
  18. 2026SCAR: Self-Supervised Continuous Action Representation Learningcited by 1
  19. 2026FlexLAM: Resolving the Bottleneck Trade-off in Latent Action Learningcited by 1
  20. 2026Improving Cross-embodiment Transfer in Latent Action Models with Action-Similarity Supervisioncited by 1
  21. 2026PoLAR: Factorizing Extent and Mode in Latent Actions for Robot Policy Learningcited by 0
  22. 2026Latent Actions from Factorized Transition Effects under Agent Ambiguitycited by 0
  23. 2026Latent-Action-Guided Video-Language Feature Learning for Surgical Instrument-Tissue Interaction Recognitioncited by 0
  24. 2026Reconstructing Is Not Acting: Action-Centric Latent Dynamics Modelingcited by 0
  25. 2026Algebraic Consistency Alone Does Not Certify Temporal Structure in Latent Action Modelscited by 0
  26. 2026Less Language, More Latents: Annotation-Efficient VLAs for Drivingcited by 0
  27. 2026Predictive Self-Supervised Learning Provably Identifies Stochastic Signals under Nuisancecited by 0
  28. 2026ATI-VLA: Action-Centric Predictive Vision-Language-Action Models via Actionable Alignment Then Adaptive Injectioncited by 0
  29. 2026Native Action-Prior Learning from Videos for World Action Modelscited by 0
  30. 2026TERRA: Learning Transportable Latent Actions through Temporal Effect Representation and Relational Alignmentcited by 0
  31. 2026OneLA: Learning Many Worlds from Latent Actionscited by 0
  32. 2026VITAL: From Visual Motion to Precise Actioncited by 0