# Latent Actions from Factorized Transition Effects under Agent Ambiguity

arXiv: https://arxiv.org/abs/2606.30544 · HTML: https://mlsanity.com/paper/2606.30544/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 35 of its 42 references that are in the map.
- First posted on arXiv 29 Jun 2026.

## Builds on (most cited first, 30 of 35)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [TD-MPC2: Scalable, Robust World Models for Continuous Control](https://mlsanity.com/paper/2310.16828/) (2023)
- [Variational Autoencoders and Nonlinear ICA: A Unifying Framework](https://mlsanity.com/paper/1907.04809/) (2019)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning](https://mlsanity.com/paper/2411.04983/) (2024)
- [Latent Action Pretraining from Videos](https://mlsanity.com/paper/2410.11758/) (2024)
- [Generating Holistic 3D Human Motion from Speech](https://mlsanity.com/paper/2212.04420/) (2022)
- [Learning to Act without Actions](https://mlsanity.com/paper/2312.10812/) (2023)
- [Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wild](https://mlsanity.com/paper/2406.09905/) (2024)
- [The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixels](https://mlsanity.com/paper/2101.02722/) (2021)
- [Learning Latent Action World Models In The Wild](https://mlsanity.com/paper/2601.05230/) (2026)
- [Latent Action Learning Requires Supervision in the Presence of Distractors](https://mlsanity.com/paper/2502.00379/) (2025)
- [Guess What Moves: Unsupervised Video and Image Segmentation by Anticipating Motion](https://mlsanity.com/paper/2205.07844/) (2022)
- [Image Compression with Product Quantized Masked Image Modeling](https://mlsanity.com/paper/2212.07372/) (2022)
- [Unsupervised Image Representation Learning with Deep Latent Particles](https://mlsanity.com/paper/2205.15821/) (2022)
- [LAOF: Robust Latent Action Learning with Optical Flow Constraints](https://mlsanity.com/paper/2511.16407/) (2025)
- [X-Dancer: Expressive Music to Human Dance Video Generation](https://mlsanity.com/paper/2502.17414/) (2025)
- [Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modeling](https://mlsanity.com/paper/2603.04553/) (2026)
- [Object-Centric Latent Action Learning](https://mlsanity.com/paper/2502.09680/) (2025)
- [MotionStrata: Hierarchical Motion Latents for Compact Video Autoencoding](https://mlsanity.com/paper/2506.07136/) (2025)
- [Un-EVIMO: Unsupervised Event-Based Independent Motion Segmentation](https://mlsanity.com/paper/2312.00114/) (2023)
- [Learning Additively Compositional Latent Actions for Embodied AI](https://mlsanity.com/paper/2604.03340/) (2026)
- [Segment to Focus: Guiding Latent Action Models in the Presence of Distractors](https://mlsanity.com/paper/2602.02259/) (2026)
- [Articulation in Motion: Prior-free Part Mobility Analysis for Articulated Objects By Dynamic-Static Disentanglement](https://mlsanity.com/paper/2603.02910/) (2026)

## Abstract (arXiv)

Latent Action Models (LAMs) learn action-like proxies from observation. However, in multi-object or distractor-rich scenes, observations contain not only agent motion but also distractors, camera dynamics, and background changes, making recovery of the underlying action intrinsically ambiguous without supervision. We argue that the appropriate unsupervised target is therefore not the true action itself, but a state-conditioned compositional summary of the transition effects present in the scene, enabling better action alignment and more effective utilization of action supervision. To this end, we propose a two-stage framework. We first pretrain Observed Transition Factorization (OTF) to discover reusable local transition primitives using a compositional codebook. We then aggregate these primitives into compact state-conditioned latent actions, instantiated as OTF-LAM-Pixel under the standard inverse-forward dynamics framework and OTF-LAM-Dino, a decoder-free variant operating in a frozen DINOv2 representation space. Experiments show that the learned transition primitives transfer across visual appearance and morphology, while the resulting latent actions exhibit substantially stronger action alignment than existing LAMs, make more effective use of downstream action supervision, and achieve competitive or superior downstream policy performance.