# Developing Vision-Language-Action Model from Egocentric Videos arXiv: https://arxiv.org/abs/2509.21986 · HTML: https://mlsanity.com/paper/2509.21986/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Placed in Human-video and latent-action pretraining, one of 13 directions in Vision-language-action models. - Builds on 49 of its 60 references that are in the map. - First posted on arXiv 26 Sep 2025. ## Placed in - Vision-language-action models / [Human-video and latent-action pretraining](https://mlsanity.com/vla/human-video-and-latent-action-pretraining/) ## Builds on (most cited first, 30 of 49) - [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022) - [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017) - [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024) - [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023) - [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023) - [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024) - [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024) - [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022) - [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023) - [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021) - [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024) - [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025) - [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017) - [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018) - [PaliGemma: A versatile 3B VLM for transfer](https://mlsanity.com/paper/2407.07726/) (2024) - [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022) - [FAST: Efficient Action Tokenization for Vision-Language-Action Models](https://mlsanity.com/paper/2501.09747/) (2025) - [BridgeData V2: A Dataset for Robot Learning at Scale](https://mlsanity.com/paper/2308.12952/) (2023) - [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020) - [SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics](https://mlsanity.com/paper/2506.01844/) (2025) - [BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning](https://mlsanity.com/paper/2202.02005/) (2022) - [Evaluating Real-World Robot Manipulation Policies in Simulation](https://mlsanity.com/paper/2405.05941/) (2024) - [AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems](https://mlsanity.com/paper/2503.06669/) (2025) - [HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction](https://mlsanity.com/paper/2203.01577/) (2022) - [Latent Action Pretraining from Videos](https://mlsanity.com/paper/2410.11758/) (2024) - [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022) - [Rectified Flow: A Marginal Preserving Approach to Optimal Transport](https://mlsanity.com/paper/2209.14577/) (2022) - [H2O: Two Hands Manipulating Objects for First Person Interaction Recognition](https://mlsanity.com/paper/2104.11181/) (2021) - [Affordances from Human Videos as a Versatile Representation for Robotics](https://mlsanity.com/paper/2304.08488/) (2023) - [MimicPlay: Long-Horizon Imitation Learning by Watching Human Play](https://mlsanity.com/paper/2302.12422/) (2023) ## Built on by (most cited first, 11 of 11) - [A Survey on Efficient Vision-Language-Action Models](https://mlsanity.com/paper/2510.24795/) (2025) - [Egocentric World Model for Photorealistic Hand-Object Interaction Synthesis](https://mlsanity.com/paper/2603.13615/) (2026) - [ActiveMimic: Egocentric Video Pretraining with Active Perception](https://mlsanity.com/paper/2606.06194/) (2026) - [Robot Learning from Human Videos: A Survey](https://mlsanity.com/paper/2604.27621/) (2026) - [From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Data](https://mlsanity.com/paper/2606.00054/) (2026) - [Flow as Flow: Modeling Robot Velocity Fields as Probability Velocity Fields for Flow-Based Object Manipulation](https://mlsanity.com/paper/2606.23090/) (2026) - [Hand-Object Interaction in the Age of Large Foundation Models:Reconstruction, Generation, and Embodied Transfer](https://mlsanity.com/paper/2607.28394/) (2026) - [UniWAM: Unified World-Action Model](https://mlsanity.com/paper/2610.02054/) (2026) - [MotionBits: Video Segmentation through Motion-Level Analysis of Rigid Bodies](https://mlsanity.com/paper/2603.06846/) (2026) - [AFUN: Towards an Affordance Foundation Model for Functionality Understanding](https://mlsanity.com/paper/2606.02551/) (2026) - [Exo2EgoPose: Leveraging Exocentric Demonstrations for Vision-Language guided Egocentric 3D Hand Pose Forecasting](https://mlsanity.com/paper/2607.15890/) (2026) ## Abstract (arXiv) Egocentric videos capture how humans manipulate objects and tools, providing diverse motion cues for learning object manipulation. Unlike the costly, expert-driven manual teleoperation commonly used in training Vision-Language-Action models (VLAs), egocentric videos offer a scalable alternative. However, prior studies that leverage such videos for training robot policies typically rely on auxiliary annotations, such as detailed hand-pose recordings. Consequently, it remains unclear whether VLAs can be trained directly from raw egocentric videos. In this work, we address this challenge by leveraging EgoScaler, a framework that extracts 6DoF object manipulation trajectories from egocentric videos without requiring auxiliary recordings. We apply EgoScaler to four large-scale egocentric video datasets and automatically refine noisy or incomplete trajectories, thereby constructing a new large-scale dataset for VLA pre-training. Our experiments with a state-of-the-art π0 architecture in both simulated and real-robot environments yield three key findings: (i) pre-training on our dataset improves task success rates by over 20% compared to training from scratch, (ii) the performance is competitive with that achieved using real-robot datasets, and (iii) combining our dataset with real-robot data yields further improvements. These results demonstrate that egocentric videos constitute a promising and scalable resource for advancing VLA research.