# LAOF: Robust Latent Action Learning with Optical Flow Constraints

arXiv: https://arxiv.org/abs/2511.16407 · HTML: https://mlsanity.com/paper/2511.16407/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2025.

## Built on by (most cited first, 16 of 16)

- [VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model](https://mlsanity.com/paper/2602.10098/) (2026)
- [The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook](https://mlsanity.com/paper/2604.02029/) (2026)
- [ProgressVLA: Progress-Guided Diffusion Policy for Vision-Language Robotic Manipulation](https://mlsanity.com/paper/2603.27670/) (2026)
- [RotVLA: Rotational Latent Action for Vision-Language-Action Model](https://mlsanity.com/paper/2605.13403/) (2026)
- [Learning Additively Compositional Latent Actions for Embodied AI](https://mlsanity.com/paper/2604.03340/) (2026)
- [Segment to Focus: Guiding Latent Action Models in the Presence of Distractors](https://mlsanity.com/paper/2602.02259/) (2026)
- [From Pixels to Tokens: A Systematic Study of Latent Action Supervision for Vision-Language-Action Models](https://mlsanity.com/paper/2605.04678/) (2026)
- [What Matters for Latent Actions in Robot Learning](https://mlsanity.com/paper/2608.19613/) (2026)
- [Why Latent Actions Fail, and How to Prevent It](https://mlsanity.com/paper/2605.20223/) (2026)
- [LAFP: Preserving Latent Action Structure in Latent Policy Learning via Flow Matching](https://mlsanity.com/paper/2606.10517/) (2026)
- [Latent Actions from Factorized Transition Effects under Agent Ambiguity](https://mlsanity.com/paper/2606.30544/) (2026)
- [GeoLAM: Learning Geometry-Grounded Latent Actions from Unlabeled Human Videos](https://mlsanity.com/paper/2609.17099/) (2026)
- [Devol-ONE: One Autoregressive Mixture of Transformers to Unify Vision-Language-Action and Latent World Modeling](https://mlsanity.com/paper/2609.32193/) (2026)
- [LAPT-Scale: Scaling Embodied Policies with Latent Action Pre-Training on Human Videos](https://mlsanity.com/paper/2610.13732/) (2026)
- [UniLAP: A Unified Latent Action Policy for Cross-Embodiment Manipulation](https://mlsanity.com/paper/2610.16969/) (2026)
- [OneLA: Learning Many Worlds from Latent Actions](https://mlsanity.com/paper/2610.20160/) (2026)

## Abstract (arXiv)

Learning latent actions from large-scale videos is crucial for the pre-training of scalable embodied foundation models, yet existing methods often struggle with action-irrelevant distractors. Although incorporating action supervision can alleviate these distractions, its effectiveness is restricted by the scarcity of available action labels. Optical flow represents pixel-level motion between consecutive frames, naturally suppressing background elements and emphasizing moving objects. Motivated by this, we propose robust Latent Action learning with Optical Flow constraints, called LAOF, a pseudo-supervised framework that leverages the agent's optical flow as an action-driven signal to learn latent action representations robust to distractors. Experimental results show that the latent representations learned by LAOF outperform existing methods on downstream imitation learning and reinforcement learning tasks. This superior performance arises from optical flow constraints, which substantially stabilize training and improve the quality of latent representations under extremely label-scarce conditions, while remaining effective as the proportion of action labels increases to 10 percent. Importantly, even without action supervision, LAOF matches or surpasses action-supervised methods trained with 1 percent of action labels.