mlsanityData to 7 Oct 2026

Vision-Language Models Unlock Task-Centric Latent Actions

arXiv 2601.22714 ↗First posted 30 Jan 2026cs.LG cs.AI cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Latent Action Models (LAMs) have rapidly gained traction as an important component in the pre-training pipelines of leading Vision-Language-Action models. However, they fail when observations contain action-correlated distractors, often encoding noise instead of meaningful latent actions. Humans, on the other hand, can effortlessly distinguish task-relevant motions from irrelevant details in any video given only a brief task description. In this work, we propose to utilize the common-sense reasoning abilities of Vision-Language Models (VLMs) to provide promptable representations, effectively separating controllable changes from the noise in unsupervised way. We use these representations as targets during LAM training and benchmark a wide variety of popular VLMs, revealing substantial variation in the quality of promptable representations as well as their robustness to different prompts and hyperparameters. Interestingly, we find that more recent VLMs may perform worse than older ones. Finally, we show that simply asking VLMs to ignore distractors can substantially improve latent action quality, yielding up to a six-fold increase in downstream success rates on Distracting MetaWorld.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 57 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

54 of 57 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Vision-Language Models Provide Promptable Representations for Reinforcement Learning5 Feb

Other references · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2025Qwen3 Technical Reportcited by 8,195
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2017Neural Discrete Representation Learningcited by 4,650
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2025Gemma 3 Technical Reportcited by 1,686
38 more
  1. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  2. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  3. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  4. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  6. 2024Genie: Generative Interactive Environmentscited by 706
  7. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  8. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 390
  9. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  10. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  11. 2024Latent Action Pretraining from Videoscited by 302
  12. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 222
  13. 2024Pixtral 12Bcited by 174
  14. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 168
  15. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  16. 2025Cosmos-Reason1: From Physical Common Sense To Embodied Reasoningcited by 159
  17. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  18. 2024E5-V: Universal Embeddings with Multimodal Large Language Modelscited by 123
  19. 2023Learning to Act without Actionscited by 116
  20. 2025EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videoscited by 116
  21. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 110
  22. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 108
  23. 2018Imitating Latent Policies from Observationcited by 101
  24. 2021The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixelscited by 97
  25. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  26. 2019Recent Advances in Imitation Learning from Observationcited by 92
  27. 2025Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMscited by 73
  28. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AIcited by 68
  29. 2025OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extractioncited by 54
  30. 2024DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Controlcited by 43
  31. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  32. 2025Latent Action Learning Requires Supervision in the Presence of Distractorscited by 40
  33. 2025CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrationscited by 39
  34. 2025CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learningcited by 37
  35. 2025What Do Latent Action Models Actually Learn?cited by 30
  36. 2022Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectoriescited by 27
  37. 2018Learning what you can do before doing anythingcited by 26
  38. 2025Lost in Embeddings: Information Loss in Vision-Language Modelscited by 18

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstructioncited by 11

Similar, not cited

Close in wording, no citation either way

  1. 2026From Pixels to Tokens: A Systematic Study of Latent Action Supervision for Vision-Language-Action Models
  2. 2025Seeing Space and Motion: Enhancing Latent Actions with Geometric and Dynamic Awareness for Vision-Language-Action Models
  3. 2026LARA: Latent Action Representation Alignment for Vision-Language-Action Models
  4. 2026Disentangling Spurious Correlations in Vision-Language-Action Models via Predicting Domain-Invariant Latent Lookahead
  5. 2025Don't Blind Your VLA: Aligning Visual Representations for OOD Generalization
  6. 2026LangForce: Bayesian Decomposition of Vision Language Action Models via Latent Action Queries
  7. 2026PAIR: Bridging Perception and Action in Vision-Language-Action Models
  8. 2026Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation