# CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videos

arXiv: https://arxiv.org/abs/2601.04061 · HTML: https://mlsanity.com/paper/2601.04061/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Human-video and latent-action pretraining, one of 13 directions in Vision-language-action models.
- Builds on 68 of its 76 references that are in the map.
- First posted on arXiv 7 Jan 2026.

## Placed in

- Vision-language-action models / [Human-video and latent-action pretraining](https://mlsanity.com/vla/human-video-and-latent-action-pretraining/)

## Builds on (most cited first, 30 of 68)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success](https://mlsanity.com/paper/2502.19645/) (2025)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://mlsanity.com/paper/2310.08864/) (2023)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)
- [PaliGemma: A versatile 3B VLM for transfer](https://mlsanity.com/paper/2407.07726/) (2024)
- [FAST: Efficient Action Tokenization for Vision-Language-Action Models](https://mlsanity.com/paper/2501.09747/) (2025)
- [BridgeData V2: A Dataset for Robot Learning at Scale](https://mlsanity.com/paper/2308.12952/) (2023)

## Built on by (most cited first, 30 of 31)

- [π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities](https://mlsanity.com/paper/2604.15483/) (2026)
- [VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model](https://mlsanity.com/paper/2602.10098/) (2026)
- [The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook](https://mlsanity.com/paper/2604.02029/) (2026)
- [World Guidance: World Modeling in Condition Space for Action Generation](https://mlsanity.com/paper/2602.22010/) (2026)
- [DM0: An Embodied-Native Vision-Language-Action Model towards Physical AI](https://mlsanity.com/paper/2602.14974/) (2026)
- [Wall-OSS-0.5 Technical Report](https://mlsanity.com/paper/2605.30877/) (2026)
- [RotVLA: Rotational Latent Action for Vision-Language-Action Model](https://mlsanity.com/paper/2605.13403/) (2026)
- [X-Tokenizer: A Multimodal Action Tokenizer for Vision-Language-Action Pretraining](https://mlsanity.com/paper/2606.14752/) (2026)
- [Data Pyramid for Embodied Manipulation: A Survey](https://mlsanity.com/paper/2607.24744/) (2026)
- [World-Action Models for Robot Learning and Control: A Survey](https://mlsanity.com/paper/2609.16074/) (2026)
- [Robot Learning from Human Videos: A Survey](https://mlsanity.com/paper/2604.27621/) (2026)
- [Policy-Guided World Model Planning for Language-Conditioned Visual Navigation](https://mlsanity.com/paper/2603.25981/) (2026)
- [What Matters for Latent Actions in Robot Learning](https://mlsanity.com/paper/2608.19613/) (2026)
- [Interact3D: Compositional 3D Generation of Interactive Objects](https://mlsanity.com/paper/2603.16085/) (2026)
- [SAFE-Pruner: Semantic Attention-Guided Future-Aware Token Pruning for Efficient Vision-Language-Action Manipulation](https://mlsanity.com/paper/2605.29662/) (2026)
- [From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Data](https://mlsanity.com/paper/2606.00054/) (2026)
- [Towards Predictive, Aligned, and Scalable Robot Learning](https://mlsanity.com/paper/2607.11270/) (2026)
- [LD4WAM: Learning Latent Dynamics from Human Videos for World Action Models](https://mlsanity.com/paper/2608.22403/) (2026)
- [AnyWorld: Factorized Egocentric World Models for Cross-Embodiment Generalization](https://mlsanity.com/paper/2608.29242/) (2026)
- [MuseVLA: An Adaptive Multimodal Sensing Vision-Language-Action Model for Robotic Manipulation](https://mlsanity.com/paper/2606.17598/) (2026)
- [Uncertainty-Aware World Model for Aerial Image-Goal Navigation](https://mlsanity.com/paper/2608.05597/) (2026)
- [EgoKit: Towards Unified Low-Cost Egocentric Data Collection with Heterogeneous Devices](https://mlsanity.com/paper/2605.16797/) (2026)
- [Hand-Object Interaction in the Age of Large Foundation Models:Reconstruction, Generation, and Embodied Transfer](https://mlsanity.com/paper/2607.28394/) (2026)
- [How Should Vision-Language-Action Models Use Proprioceptive State?](https://mlsanity.com/paper/2608.03052/) (2026)
- [Contrastive Action-Image Pre-training for Visuomotor Control](https://mlsanity.com/paper/2606.17256/) (2026)
- [EgoRecovery: Acquiring Failure Recovery Ability Through Human Recovery Demonstration](https://mlsanity.com/paper/2607.19745/) (2026)
- [Decoupling Intention from Trajectory: A Representational Deduction Framework for World Action Models](https://mlsanity.com/paper/2608.06994/) (2026)
- [WLA3: World Latent Action Modeling for Semantics, Dynamics, and Kinematics](https://mlsanity.com/paper/2609.15870/) (2026)
- [Algebraic Consistency Alone Does Not Certify Temporal Structure in Latent Action Models](https://mlsanity.com/paper/2609.23478/) (2026)
- [LAPT-Scale: Scaling Embodied Policies with Latent Action Pre-Training on Human Videos](https://mlsanity.com/paper/2610.13732/) (2026)

## Abstract (arXiv)

Generalist Vision-Language-Action models remain constrained by the scarcity of robotic data relative to the abundance of human video demonstrations. Existing Latent Action Models attempt to use video data but often suffer from visual entanglement, encoding noise rather than manipulation skills. To address this limitation, we propose Contrastive Latent Action Pretraining (CLAP), a framework that first uses Act-VAE to learn an executable action-token vocabulary from robot trajectories and then aligns human visual transitions with this vocabulary through contrastive learning. This alignment maps unlabeled human videos into a physically grounded latent action space rather than reconstructing appearance. Building on the aligned tokens, we train CLAP-NTP as an autoregressive VLA using robot demonstrations and pseudo-labeled human videos, preserving instruction following and object generalization. For deployment and target-domain adaptation, we further introduce a post-training strategy that combines CLAP-RF, a Rectified Flow action head for low-latency continuous action chunk prediction, with Knowledge Matching regularization to preserve pretrained semantic knowledge during fine-tuning. Extensive experiments show that CLAP achieves strong performance against competitive baselines while enabling effective skill transfer from human videos to robotic execution.