mlsanityData to 7 Oct 2026

Latent Action Pretraining from Videos

arXiv 2410.11758 ↗First posted 15 Oct 2024cs.RO cs.CL cs.CVcited by 302

On the map: Vision-language-action models → Human-video and latent-action pretraining

Abstract (arXiv)

We introduce Latent Action Pretraining for general Action models (LAPA), an unsupervised method for pretraining Vision-Language-Action (VLA) models without ground-truth robot action labels. Existing Vision-Language-Action models require action labels typically collected by human teleoperators during pretraining, which significantly limits possible data sources and scale. In this work, we propose a method to learn from internet-scale videos that do not have robot action labels. We first train an action quantization model leveraging VQ-VAE-based objective to learn discrete latent actions between image frames, then pretrain a latent VLA model to predict these latent actions from observations and task descriptions, and finally finetune the VLA on small-scale robot manipulation data to map from latent to robot actions. Experimental results demonstrate that our method significantly outperforms existing techniques that train robot manipulation policies from large-scale videos. Furthermore, it outperforms the state-of-the-art VLA model trained with robotic action labels on real-world manipulation tasks that require language conditioning, generalization to unseen objects, and semantic generalization to unseen instructions. Training only on human manipulation videos also shows positive transfer, opening up the potential for leveraging web-scale data for robotics foundation model.

Key facts. Cited by 302 papers in the map (data through 7 Oct 2026). Placed in Human-video and latent-action pretraining, one of 13 directions in Vision-language-action models. Builds on 58 of its 63 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

58 of 63 references.

References · 58

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2017Neural Discrete Representation Learningcited by 4,650
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  10. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  11. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  12. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
38 more
  1. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  2. 2024Genie: Generative Interactive Environmentscited by 706
  3. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  4. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  5. 2020Rescaling Egocentric Visioncited by 529
  6. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  7. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  8. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  9. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  10. 2023Learning Interactive Real-World Simulatorscited by 440
  11. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  12. 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 340
  13. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  14. 2019Learning Latent Plans from Playcited by 310
  15. 2024Diffusion Models Are Real-Time Game Enginescited by 282
  16. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  17. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  18. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  19. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  20. 2024RT-H: Action Hierarchies Using Languagecited by 204
  21. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 194
  22. 2023Video Language Planningcited by 160
  23. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 159
  24. 2024Behavior Generation with Latent Actionscited by 159
  25. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  26. 2023Learning to Act without Actionscited by 116
  27. 2018Imitating Latent Policies from Observationcited by 101
  28. 2022VideoDex: Learning Dexterity from Internet Videoscited by 101
  29. 2024LLARVA: Vision-Action Instruction Tuning Enhances Robot Learningcited by 87
  30. 2024Video as the New Language for Real-World Decision Makingcited by 86
  31. 2024Dreamitate: Real-World Visuomotor Policy Learning via Video Generationcited by 84
  32. 2024EquiBot: SIM(3)-Equivariant Diffusion Policy for Generalizable and Data Efficient Learningcited by 72
  33. 2024LLaRA: Supercharging Robot Learning Data for Vision-Language Policycited by 51
  34. 2024QueST: Self-Supervised Skill Abstractions for Learning Continuous Controlcited by 51
  35. 2023Zero-Shot Robot Manipulation from Passive Human Videoscited by 49
  36. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 46
  37. 2023An Unbiased Look at Datasets for Visuo-Motor Pre-Trainingcited by 41
  38. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 116; 2026: 180.

302 papers.

Sort:

By area:

  1. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  2. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 643
  3. 2025RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulationcited by 477
  4. 2026World Action Models are Zero-shot Policiescited by 411
  5. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 390
  6. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  7. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 285
  8. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 279
  9. 2025Motus: A Unified Latent Action World Modelcited by 231
  10. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasetscited by 218
  11. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 168
  12. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
38 more
  1. 2025mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAscited by 133
  2. 2025Unified Vision-Language-Action Modelcited by 128
  3. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 127
  4. 2025GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Datacited by 120
  5. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 117
  6. 2025EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videoscited by 116
  7. 2025GR-3 Technical Reportcited by 110
  8. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 108
  9. 2025FLARE: Robot Learning with Implicit World Modelingcited by 105
  10. 2026VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Modelcited by 104
  11. 2026StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developingcited by 103
  12. 2025Discrete Diffusion VLA: Bringing Discrete Diffusion to Action Decoding in Vision-Language-Action Policiescited by 90
  13. 2024What Matters in Building Vision-Language-Action Models for Generalist Robotscited by 88
  14. 2025DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Graspingcited by 87
  15. 2025InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policycited by 83
  16. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  17. 2025Universal Actions for Enhanced Embodied Foundation Modelscited by 80
  18. 2025InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulationcited by 71
  19. 2026DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Controlcited by 71
  20. 2025RynnVLA-002: A Unified Vision-Language-Action and World Modelcited by 70
  21. 2026Being-H0.7: A Latent World-Action Model from Egocentric Videoscited by 68
  22. 2025VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizerscited by 67
  23. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  24. 2024Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videoscited by 65
  25. 2025WorldEval: World Model as Real-World Robot Policies Evaluatorcited by 64
  26. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 63
  27. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 61
  28. 2025EnerVerse: Envisioning Embodied Future Space for Robotics Manipulationcited by 59
  29. 2025villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Modelscited by 58
  30. 2026EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the Worldcited by 49
  31. 2025AutoEval: Autonomous Evaluation of Generalist Robot Manipulation Policies in the Real Worldcited by 46
  32. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  33. 2026LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policiescited by 46
  34. 2025VideoWorld: Exploring Knowledge Learning from Unlabeled Videoscited by 42
  35. 2026Learning Latent Action World Models In The Wildcited by 42
  36. 2025Latent Action Learning Requires Supervision in the Presence of Distractorscited by 40
  37. 2025LeVERB: Humanoid Whole-Body Control with Latent Vision-Language Instructioncited by 40
  38. 2025CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrationscited by 39

50 most cited of 302 All 302 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026LA4VLA: Learning to Act without Seeing via Language-Action Pretraining
  2. 2025RynnVLA-001: Using Human Demonstrations to Improve Robot Manipulation
  3. 2025Enhancing Generalization in Vision-Language-Action Models by Preserving Pretrained Representations
  4. 2026ViDAL: A Visual Dynamics-Grounded Action Latent Space for Vision-Language-Action Models
  5. 2025LoLA: Long Horizon Latent Action Learning for General Robot Manipulation
  6. 2026QuoVLA: Quotient Space for Vision-Language-Action Models
  7. 2026VLAFlow: A Unified Training Framework for Vision-Language-Action Models via Co-training and Future Latent Alignment
  8. 2026Learning to Move Before Learning to Do: Task-Agnostic pretraining for VLAs