# UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models

arXiv: https://arxiv.org/abs/2602.20231 · HTML: https://mlsanity.com/paper/2602.20231/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Human-video and latent-action pretraining, one of 13 directions in Vision-language-action models.
- Builds on 47 of its 50 references that are in the map.
- First posted on arXiv 23 Feb 2026.

## Placed in

- Vision-language-action models / [Human-video and latent-action pretraining](https://mlsanity.com/vla/human-video-and-latent-action-pretraining/)

## Builds on (most cited first, 30 of 47)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)
- [Genie: Generative Interactive Environments](https://mlsanity.com/paper/2402.15391/) (2024)
- [CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks](https://mlsanity.com/paper/2112.03227/) (2021)
- [SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model](https://mlsanity.com/paper/2501.15830/) (2025)
- [CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models](https://mlsanity.com/paper/2503.22020/) (2025)
- [CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation](https://mlsanity.com/paper/2411.19650/) (2024)
- [UniVLA: Learning to Act Anywhere with Task-centric Latent Actions](https://mlsanity.com/paper/2505.06111/) (2025)
- [AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems](https://mlsanity.com/paper/2503.06669/) (2025)
- [Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation](https://mlsanity.com/paper/2312.13139/) (2023)
- [Vision-Language Foundation Models as Effective Robot Imitators](https://mlsanity.com/paper/2311.01378/) (2023)
- [3D-VLA: A 3D Vision-Language-Action Generative World Model](https://mlsanity.com/paper/2403.09631/) (2024)
- [Latent Action Pretraining from Videos](https://mlsanity.com/paper/2410.11758/) (2024)
- [GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation](https://mlsanity.com/paper/2410.06158/) (2024)
- [Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models](https://mlsanity.com/paper/2310.10639/) (2023)
- [Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets](https://mlsanity.com/paper/2504.02792/) (2025)
- [Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models](https://mlsanity.com/paper/2502.19417/) (2025)
- [What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Data](https://mlsanity.com/paper/2204.06252/) (2022)
- [HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation](https://mlsanity.com/paper/2502.05485/) (2025)

## Built on by (most cited first, 5 of 5)

- [The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook](https://mlsanity.com/paper/2604.02029/) (2026)
- [RotVLA: Rotational Latent Action for Vision-Language-Action Model](https://mlsanity.com/paper/2605.13403/) (2026)
- [Robot Learning from Human Videos: A Survey](https://mlsanity.com/paper/2604.27621/) (2026)
- [LAWM-3D: Learning 3D-Aware Latent Actions from Human Videos for Generalizable Robot World Models](https://mlsanity.com/paper/2608.05706/) (2026)
- [EgoRecovery: Acquiring Failure Recovery Ability Through Human Recovery Demonstration](https://mlsanity.com/paper/2607.19745/) (2026)

## Abstract (arXiv)

Latent action representations learned from unlabeled videos have recently emerged as a promising paradigm for pretraining vision-language-action (VLA) models without explicit robot action supervision. However, latent actions derived solely from RGB observations primarily encode appearance-driven dynamics and lack explicit 3D geometric structure, which is essential for precise and contact-rich manipulation. To address this limitation, we introduce UniLACT, a transformer-based VLA model that incorporates geometric structure through depth-aware latent pretraining, enabling downstream policies to inherit stronger spatial priors. To facilitate this process, we propose UniLARN, a unified latent action learning framework based on inverse and forward dynamics objectives that learns a shared embedding space for RGB and depth while explicitly modeling their cross-modal interactions. This formulation produces modality-specific and unified latent action representations that serve as pseudo-labels for the depth-aware pretraining of UniLACT. Extensive experiments in both simulation and real-world settings demonstrate the effectiveness of depth-aware unified latent action representations. UniLACT consistently outperforms RGB-based latent action baselines under in-domain and out-of-domain pretraining regimes, as well as on both seen and unseen manipulation tasks.The project page is at https://manishgovind.github.io/unilact-vla/