# Learning to Act Robustly with View-Invariant Latent Actions

arXiv: https://arxiv.org/abs/2601.02994 · HTML: https://mlsanity.com/paper/2601.02994/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 6 Jan 2026.

## Builds on (most cited first, 23 of 23)

- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [BridgeData V2: A Dataset for Robot Learning at Scale](https://mlsanity.com/paper/2308.12952/) (2023)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [UniVLA: Learning to Act Anywhere with Task-centric Latent Actions](https://mlsanity.com/paper/2505.06111/) (2025)
- [MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations](https://mlsanity.com/paper/2310.17596/) (2023)
- [Latent Action Pretraining from Videos](https://mlsanity.com/paper/2410.11758/) (2024)
- [Soft-DTW: a Differentiable Loss Function for Time-Series](https://mlsanity.com/paper/1703.01541/) (2017)
- [RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot](https://mlsanity.com/paper/2307.00595/) (2023)
- [AdaWorld: Learning Adaptable World Models with Latent Actions](https://mlsanity.com/paper/2503.18938/) (2025)
- [villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models](https://mlsanity.com/paper/2507.23682/) (2025)
- [Latent Action Learning Requires Supervision in the Presence of Distractors](https://mlsanity.com/paper/2502.00379/) (2025)
- [VideoWorld: Exploring Knowledge Learning from Unlabeled Videos](https://mlsanity.com/paper/2501.09781/) (2025)
- [RoboHive: A Unified Framework for Robot Learning](https://mlsanity.com/paper/2310.06828/) (2023)
- [View-Invariant Policy Learning via Zero-Shot Novel View Synthesis](https://mlsanity.com/paper/2409.03685/) (2024)
- [RoboUniView: Visual-Language Model with Unified View Representation for Robotic Manipulation](https://mlsanity.com/paper/2406.18977/) (2024)
- [Latent Action Diffusion for Cross-Embodiment Manipulation](https://mlsanity.com/paper/2506.14608/) (2025)
- [Contrastive Representation Regularization for Vision-Language-Action Models](https://mlsanity.com/paper/2510.01711/) (2025)
- [Do You Know Where Your Camera Is? View-Invariant Policy Learning with Camera Conditioning](https://mlsanity.com/paper/2510.02268/) (2025)
- [CLASS: Contrastive Learning via Action Sequence Supervision for Robot Manipulation](https://mlsanity.com/paper/2508.01600/) (2025)
- [Imagination at Inference: Synthesizing In-Hand Views for Robust Visuomotor Policy Inference](https://mlsanity.com/paper/2509.15717/) (2025)
- [NVSPolicy: Adaptive Novel-View Synthesis for Generalizable Language-Conditioned Policy Learning](https://mlsanity.com/paper/2505.10359/) (2025)

## Built on by (most cited first, 8 of 8)

- [The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook](https://mlsanity.com/paper/2604.02029/) (2026)
- [From Fixed to Free Cameras: Calibration-Free View-Robust Vision-Language-Action Model](https://mlsanity.com/paper/2607.05396/) (2026)
- [LAFP: Preserving Latent Action Structure in Latent Policy Learning via Flow Matching](https://mlsanity.com/paper/2606.10517/) (2026)
- [Selective Cross-View Consistency for World Action Models: Held-Out Viewpoint Robustness Without Test-Time Camera Information](https://mlsanity.com/paper/2608.21402/) (2026)
- [SCAR: Self-Supervised Continuous Action Representation Learning](https://mlsanity.com/paper/2605.16412/) (2026)
- [Improving Cross-embodiment Transfer in Latent Action Models with Action-Similarity Supervision](https://mlsanity.com/paper/2609.19846/) (2026)
- [PoLAR: Factorizing Extent and Mode in Latent Actions for Robot Policy Learning](https://mlsanity.com/paper/2606.21139/) (2026)
- [OneLA: Learning Many Worlds from Latent Actions](https://mlsanity.com/paper/2610.20160/) (2026)

## Abstract (arXiv)

Vision-based robotic policies often struggle with even minor viewpoint changes, underscoring the need for view-invariant visual representations. This challenge becomes more pronounced in real-world settings, where viewpoint variability is unavoidable and can significantly disrupt policy performance. Existing methods typically learn invariance from multi-view observations at the scene level, but such approaches rely on visual appearance and fail to incorporate the physical dynamics essential for robust generalization. We propose View-Invariant Latent Action (VILA), which models a latent action capturing transition patterns across trajectories to learn view-invariant representations grounded in physical dynamics. VILA aligns these latent actions across viewpoints using an action-guided objective based on ground-truth action sequences. Experiments in both simulation and the real world show that VILA-based policies generalize effectively to unseen viewpoints and transfer well to new tasks, establishing VILA as a strong pretraining framework that improves robustness and downstream learning performance.