# LAWM-3D: Learning 3D-Aware Latent Actions from Human Videos for Generalizable Robot World Models

arXiv: https://arxiv.org/abs/2608.05706 · HTML: https://mlsanity.com/paper/2608.05706/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and robot world models, one of 11 directions in World models.
- Builds on 23 of its 48 references that are in the map.
- First posted on arXiv 6 Aug 2026.

## Placed in

- World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/)

## Builds on (most cited first, 23 of 23)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)
- [AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems](https://mlsanity.com/paper/2503.06669/) (2025)
- [EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video](https://mlsanity.com/paper/2505.11709/) (2025)
- [World Simulation with Video Foundation Models for Physical AI](https://mlsanity.com/paper/2511.00062/) (2025)
- [Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulation](https://mlsanity.com/paper/2508.05635/) (2025)
- [DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos](https://mlsanity.com/paper/2602.06949/) (2026)
- [TesserAct: Learning 4D Embodied World Models](https://mlsanity.com/paper/2504.20995/) (2025)
- [WorldArena: A Unified Benchmark for Evaluating Perception and Functional Utility of Embodied World Models](https://mlsanity.com/paper/2602.08971/) (2026)
- [WoW: Towards a World omniscient World model Through Embodied Interaction](https://mlsanity.com/paper/2509.22642/) (2025)
- [GigaWorld-0: World Models as Data Engine to Empower Embodied AI](https://mlsanity.com/paper/2511.19861/) (2025)
- [CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learning](https://mlsanity.com/paper/2505.17006/) (2025)
- [Matrix-3D: Omnidirectional Explorable 3D World Generation](https://mlsanity.com/paper/2508.08086/) (2025)
- [Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Control](https://mlsanity.com/paper/2506.01943/) (2025)
- [Latent Action Pretraining Through World Modeling](https://mlsanity.com/paper/2509.18428/) (2025)
- [WorldStereo: Bridging Camera-Guided Video Generation and Scene Reconstruction via 3D Geometric Memories](https://mlsanity.com/paper/2603.02049/) (2026)
- [MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstruction](https://mlsanity.com/paper/2602.03668/) (2026)
- [Kinema4D: Kinematic 4D World Modeling for Spatiotemporal Embodied Simulation](https://mlsanity.com/paper/2603.16669/) (2026)
- [UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models](https://mlsanity.com/paper/2602.20231/) (2026)

## Built on by (most cited first, 1 of 1)

- [GeoLAM: Learning Geometry-Grounded Latent Actions from Unlabeled Human Videos](https://mlsanity.com/paper/2609.17099/) (2026)

## Abstract (arXiv)

World models enable agents to perform forward rollout and planning without real-world interaction. However, their application in open-world embodied intelligence remains limited by the high cost of action annotations and the heterogeneity of action spaces across platforms. Recently, latent action models (LAMs) have alleviated this bottleneck by learning action representations directly from unlabeled human videos in a self-supervised manner. Nevertheless, most existing LAMs rely on single-view inputs and operate primarily in 2D pixel space, raising a fundamental question: can simply incorporating multi-view videos into LAM training endow the learned latent actions with 3D-aware perception? Our study shows that the answer is negative. The primary reasons lie in future-frame appearance leakage as well as inter-camera appearance discrepancies and viewpoint variations. To address these issues, we propose LAWM-3D, which introduces three tightly coupled key designs: (1) a multi-view invariant unified action tokenization scheme for learning 3D-aware latent actions; (2) a geometric alignment constraint that anchors intermediate encoder features to a pretrained 3D foundation model, thereby explicitly providing cross-view geometric correspondences; and (3) a non-injective RGB-D joint reconstruction objective that prevents shortcut learning from future-frame appearance information, forcing the LAM to focus supervision on motion cues with geometric significance. Importantly, these components are not simply stacked but are tightly coupled through a unified motivation. Built upon a two-stage paradigm of large-scale human video pretraining followed by robot fine-tuning, extensive experiments demonstrate that the proposed 3D-aware latent actions significantly improve world model performance, achieving SOTA results in generation quality, physical consistency, and generalization ability.