LAWM-3D: Learning 3D-Aware Latent Actions from Human Videos for Generalizable Robot World Models
On the map: World models → Embodied and robot world models
Abstract (arXiv)
World models enable agents to perform forward rollout and planning without real-world interaction. However, their application in open-world embodied intelligence remains limited by the high cost of action annotations and the heterogeneity of action spaces across platforms. Recently, latent action models (LAMs) have alleviated this bottleneck by learning action representations directly from unlabeled human videos in a self-supervised manner. Nevertheless, most existing LAMs rely on single-view inputs and operate primarily in 2D pixel space, raising a fundamental question: can simply incorporating multi-view videos into LAM training endow the learned latent actions with 3D-aware perception? Our study shows that the answer is negative. The primary reasons lie in future-frame appearance leakage as well as inter-camera appearance discrepancies and viewpoint variations. To address these issues, we propose LAWM-3D, which introduces three tightly coupled key designs: (1) a multi-view invariant unified action tokenization scheme for learning 3D-aware latent actions; (2) a geometric alignment constraint that anchors intermediate encoder features to a pretrained 3D foundation model, thereby explicitly providing cross-view geometric correspondences; and (3) a non-injective RGB-D joint reconstruction objective that prevents shortcut learning from future-frame appearance information, forcing the LAM to focus supervision on motion cues with geometric significance. Importantly, these components are not simply stacked but are tightly coupled through a unified motivation. Built upon a two-stage paradigm of large-scale human video pretraining followed by robot fine-tuning, extensive experiments demonstrate that the proposed 3D-aware latent actions significantly improve world model performance, achieving SOTA results in generation quality, physical consistency, and generalization ability.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 23 of its 48 references that are in the map. First posted on arXiv 6 Aug 2026.
Builds on
23 of 48 references.
Closest ancestors · 5
Same direction, newest first
- 2026Kinema4D: Kinematic 4D World Modeling for Spatiotemporal Embodied Simulation17 Mar
- 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos6 Feb
- 2025GigaWorld-0: World Models as Data Engine to Empower Embodied AI25 Nov
- 2025Latent Action Pretraining Through World Modeling22 Sep
- 2025TesserAct: Learning 4D Embodied World Models29 Apr
Other references · 18
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2013Auto-Encoding Variational Bayescited by 4,053
- 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
- 2017Decoupled Weight Decay Regularizationcited by 1,831
- 2018World Modelscited by 1,513
- 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
- 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 405
- 2025EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Videocited by 211
- 2025World Simulation with Video Foundation Models for Physical AIcited by 200
- 2025Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulationcited by 137
- 2026WorldArena: A Unified Benchmark for Evaluating Perception and Functional Utility of Embodied World Modelscited by 76
- 2025WoW: Towards a World omniscient World model Through Embodied Interactioncited by 67
6 more
- 2025CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learningcited by 40
- 2025Matrix-3D: Omnidirectional Explorable 3D World Generationcited by 35
- 2025Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Controlcited by 34
- 2026WorldStereo: Bridging Camera-Guided Video Generation and Scene Reconstruction via 3D Geometric Memoriescited by 18
- 2026MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstructioncited by 14
- 2026UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Modelscited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies
- 2026LD4WAM: Learning Latent Dynamics from Human Videos for World Action Models
- 2026Learning 4D Geometric Priors for Inference-Efficient World Action Models
- 2026LARY: A Latent Action Representation Yielding Benchmark for Generalizable Vision-to-Action Alignment
- 2026LaWM: Least Action World Models for Long-Horizon Physical Consistency from Visual Observations
- 2025LatBot: Distilling Universal Latent Actions for Vision-Language-Action Models
- 2025Seeing Space and Motion: Enhancing Latent Actions with Geometric and Dynamic Awareness for Vision-Language-Action Models
- 2026Spatially Aware World Action Model via Geometric Latent Diffusion