mlsanityData to 7 Oct 2026

Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos

arXiv 2507.15597 ↗First posted 21 Jul 2025cs.CV cs.LG cs.ROcited by 94

On the map: Vision-language-action models → Human-video and latent-action pretraining

Abstract (arXiv)

We introduce Being-H0, a dexterous Vision-Language-Action model (VLA) trained on large-scale human videos. Existing VLAs struggle with complex manipulation tasks requiring high dexterity and generalize poorly to novel scenarios and tasks, primarily due to their reliance on synthetic data with significant sim-to-real gaps or teleoperated demonstrations lacking scale and diversity. To address this data bottleneck, we propose leveraging human hands as a foundation manipulator, capitalizing on the rich dexterity and scalability present in web data. Our approach centers on physical instruction tuning, a novel training paradigm that combines large-scale VLA pretraining from human videos, physical space alignment for 3D reasoning, and post-training adaptation for robotic tasks. Additionally, we introduce a part-level motion tokenization method which achieves millimeter-level reconstruction accuracy to model precise hand trajectories for action learning. To support our proposed paradigm, we further develop a comprehensive data curation pipeline that integrates heterogeneous sources -- including motion capture, VR, and RGB-only videos -- into a large-scale dataset with millions of motion-based instructional instances. We empirically show the excellence of Being-H0 in hand motion generation and instruction following, and it also scales well with model and data sizes. Importantly, we observe the expected gains of Being-H0 in real-world robotic manipulation as physical instruction tuning is applied. More details are available at https://beingbeyond.github.io/Being-H0.

Key facts. Cited by 94 papers in the map (data through 7 Oct 2026). Placed in Human-video and latent-action pretraining, one of 13 directions in Vision-language-action models. Builds on 146 of its 161 references that are in the map. First posted on arXiv 21 Jul 2025.

Builds on

146 of 161 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025DexVLG: Dexterous Vision-Language-Grasp Model at Scale3 Jul
  2. 2025GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Data6 May
  3. 2025DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Grasping28 Feb

Other references · 143

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2025Qwen2.5-VL Technical Reportcited by 5,418
  10. 2017Neural Discrete Representation Learningcited by 4,650
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
38 more
  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  3. 2023Qwen Technical Reportcited by 3,273
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  6. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  9. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  10. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  11. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  12. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  13. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  14. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  15. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  16. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  17. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  18. 2022Human Motion Diffusion Modelcited by 1,062
  19. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  20. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  21. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  22. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  23. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  24. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  25. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  26. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  27. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
  28. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 643
  29. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  30. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  31. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 577
  32. 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 567
  33. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  34. 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
  35. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  36. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  37. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  38. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 390

50 most cited of 143 All 143 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 21; 2026: 73.

94 papers.

Sort:

By area:

  1. 2025VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Modelcited by 140
  2. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 117
  3. 2026Being-H0.7: A Latent World-Action Model from Egocentric Videoscited by 68
  4. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 63
  5. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 61
  6. 2026EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the Worldcited by 49
  7. 2026Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Modelscited by 47
  8. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  9. 2026World Model for Robot Learning: A Comprehensive Surveycited by 42
  10. 2026Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodimentscited by 42
  11. 2025A Survey on Efficient Vision-Language-Action Modelscited by 34
  12. 2025Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Modelcited by 33
38 more
  1. 2026ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?cited by 33
  2. 2026Native Video-Action Pretraining for Generalizable Robot Controlcited by 30
  3. 2026RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalizationcited by 29
  4. 2025SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modelingcited by 27
  5. 2026CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videoscited by 27
  6. 2026From Foundation to Application: Improving VLA Models in Practicecited by 25
  7. 2026Xiaomi-Robotics-1: Scaling Vision-Language-Action Models with over 100K Hours of Real-World Trajectoriescited by 24
  8. 2025In-N-On: Scaling Egocentric Manipulation with in-the-wild and on-task Datacited by 21
  9. 2026A Systematic Study of Data Modalities and Strategies for Co-training Large Behavior Models for Robot Manipulationcited by 20
  10. 2026FRAPPE: Infusing World Modeling into Generalist Policies via Multiple Future Representation Alignmentcited by 19
  11. 2025MergeVLA: Cross-Skill Model Merging Toward a Generalist Vision-Language-Action Agentcited by 15
  12. 2026Next Forcing: Causal World Modeling with Multi-Chunk Predictioncited by 14
  13. 2026ConLA: Contrastive Latent Action Learning from Human Videos for Robotic Manipulationcited by 11
  14. 2025Spatial-Aware VLA Pretraining through Visual-Physical Alignment from Human Videoscited by 10
  15. 2025PhysBrain: Human Egocentric Data as a Bridge from Vision Language Models to Physical Intelligencecited by 10
  16. 2026World-Language-Action Model for Unified World Modeling, Language Reasoning, and Action Synthesiscited by 10
  17. 2025Dexplore: Scalable Neural Control for Dexterous Manipulation from Reference-Scoped Explorationcited by 9
  18. 2025METIS: Multi-Source Egocentric Training for Integrated Dexterous Vision-Language-Action Modelcited by 9
  19. 2026HumanNet: Scaling Human-centric Video Learning to One Million Hourscited by 9
  20. 2026Dexora: Open-source VLA for High-DoF Bimanual Dexteritycited by 9
  21. 2026Do as I Do: Dexterous Manipulation Data from Everyday Human Videoscited by 9
  22. 2025End-to-End Dexterous Arm-Hand VLA Policies via Shared Autonomy: VR Teleoperation Augmented by Autonomous Hand VLA Policy for Efficient Data Collectioncited by 8
  23. 2026Joint-Aligned Latent Action: Towards Scalable VLA Pretraining in the Wildcited by 8
  24. 2026Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalizationcited by 8
  25. 2025Survey of Vision-Language-Action Models for Embodied Manipulationcited by 7
  26. 2025DexMan: Learning Bimanual Dexterous Manipulation from Human and Generated Videoscited by 7
  27. 2026EgoGrasp: World-Space Hand-Object Interaction Estimation from Egocentric Videoscited by 7
  28. 2025Transport Discrepancy as a Reliability Signal for Vision-Language-Action Modelscited by 6
  29. 2025Flowing from Reasoning to Motion: Learning 3D Hand Trajectory Prediction from Egocentric Human Interaction Videoscited by 6
  30. 2025Open-world Hand-Object Interaction Video Generation Based on Structure and Contact-aware Representationcited by 5
  31. 2026Open-AoE: An Open Egocentric Manipulation Dataset and Toolchain for Embodied Learningcited by 5
  32. 2025GR-Dexter Technical Reportcited by 4
  33. 2026Spatial Memory for Out-of-Vision Manipulation in Vision-Language-Actioncited by 4
  34. 2026Human-Centric Transferable Tactile Pre-Training for Dexterous Robotic Manipulationcited by 4
  35. 2026Data Pyramid for Embodied Manipulation: A Surveycited by 4
  36. 2026DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adaptercited by 3
  37. 2026AugVLA-3D: Depth-Driven Feature Augmentation for Vision-Language-Action Modelscited by 3
  38. 2026AoE: Always-on Egocentric Human Video Collection for Embodied AIcited by 3

50 most cited of 94 All 94 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025RynnVLA-001: Using Human Demonstrations to Improve Robot Manipulation
  2. 2025AR-VRM: Imitating Human Motions for Visual Robot Manipulation with Analogical Reasoning
  3. 2026Learning Human-Intention Priors from Large-Scale Human Demonstrations for Robotic Manipulation
  4. 2026Cross-Hand Latent Representation for Vision-Language-Action Models
  5. 2026CometVLA: Co-Training on an Embodied Data Pyramid towards Physical Understanding
  6. 2026HARP-VLA: Human-Robot Aligned Representation Learning for Vision-Language-Action Model
  7. 2026H-VLA: Hierarchical Vision-Language-Action Model with Key-Action Reasoning and Motion Planning in a Unified Action Space
  8. 2025Emergence of Human to Robot Transfer in Vision-Language-Action Models