# SCAR: Self-Supervised Continuous Action Representation Learning

arXiv: https://arxiv.org/abs/2605.16412 · HTML: https://mlsanity.com/paper/2605.16412/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and robot world models, one of 11 directions in World models.
- Builds on 33 of its 46 references that are in the map.
- First posted on arXiv 13 May 2026.

## Placed in

- World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/)

## Builds on (most cited first, 30 of 33)

- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025)
- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://mlsanity.com/paper/2310.08864/) (2023)
- [Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion](https://mlsanity.com/paper/2407.01392/) (2024)
- [RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation](https://mlsanity.com/paper/2506.18088/) (2025)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning](https://mlsanity.com/paper/2411.04983/) (2024)
- [Latent Action Pretraining from Videos](https://mlsanity.com/paper/2410.11758/) (2024)
- [LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels](https://mlsanity.com/paper/2603.19312/) (2026)
- [Motus: A Unified Latent Action World Model](https://mlsanity.com/paper/2512.13030/) (2025)
- [EgoMimic: Scaling Imitation Learning via Egocentric Video](https://mlsanity.com/paper/2410.24221/) (2024)
- [Training Agents Inside of Scalable World Models](https://mlsanity.com/paper/2509.24527/) (2025)
- [EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videos](https://mlsanity.com/paper/2507.12440/) (2025)
- [Learning to Act without Actions](https://mlsanity.com/paper/2312.10812/) (2023)
- [WoW: Towards a World omniscient World model Through Embodied Interaction](https://mlsanity.com/paper/2509.22642/) (2025)
- [villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models](https://mlsanity.com/paper/2507.23682/) (2025)
- [Learning Latent Action World Models In The Wild](https://mlsanity.com/paper/2601.05230/) (2026)
- [Latent Action Learning Requires Supervision in the Presence of Distractors](https://mlsanity.com/paper/2502.00379/) (2025)
- [CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrations](https://mlsanity.com/paper/2505.04999/) (2025)
- [RISE: Self-Improving Robot Policy with Compositional World Model](https://mlsanity.com/paper/2602.11075/) (2026)
- [What Do Latent Action Models Actually Learn?](https://mlsanity.com/paper/2506.15691/) (2025)
- [Cross-Entropy Is All You Need To Invert the Data Generating Process](https://mlsanity.com/paper/2410.21869/) (2024)
- [DiWA: Diffusion Policy Adaptation with World Models](https://mlsanity.com/paper/2508.03645/) (2025)
- [A Survey of Robot Manipulation in Contact](https://mlsanity.com/paper/2112.01942/) (2021)
- [In-N-On: Scaling Egocentric Manipulation with in-the-wild and on-task Data](https://mlsanity.com/paper/2511.15704/) (2025)
- [Latent Action Diffusion for Cross-Embodiment Manipulation](https://mlsanity.com/paper/2506.14608/) (2025)
- [Cross-Embodiment Robot Manipulation Skill Transfer using Latent Space Alignment](https://mlsanity.com/paper/2406.01968/) (2024)
- [Joint-Aligned Latent Action: Towards Scalable VLA Pretraining in the Wild](https://mlsanity.com/paper/2602.21736/) (2026)

## Built on by (most cited first, 1 of 1)

- [Video2Skill: From Streaming Experience to Reusable Embodied Skills](https://mlsanity.com/paper/2609.36691/) (2026)

## Abstract (arXiv)

Despite the central role of action in embodied intelligence, learning transferable action representations from visual transitions remains a fundamental challenge, particularly when world models must generalize across embodiments under limited data. We argue that action is not merely an auxiliary conditioning signal, but a distinct representational factor that decouples the controllable change from embodiment-specific actuation. In this work, we propose SCAR, a joint inverse-forward dynamics framework for learning unified action representations across embodiments from visual transitions. Built on a pretrained generative backbone, SCAR uses an inverse dynamics model (IDM) to infer latent actions from latent observation pairs and a forward dynamics model (FDM) to predict future dynamics conditioned on them. To make the latent space transferable rather than a generic visual bottleneck, we regularize the latent action posterior toward a standard Gaussian prior to limit arbitrary visual encoding, and introduce adversarial invariance to suppress embodiment- and environment-specific nuisance factors. Experiments on the Procgen and Robotwin dataset show that the learned unified latent action representation serves as a stronger conditioning interface for world modeling than embodiment-specific raw actions, yielding improved cross-embodiment low-data adaptation and cross-task transfer. Taken together, these results suggest that action can be learned as a shared representation of controllable change across embodiments, providing an interface for more transferable and generalizable world models.