# Dynamics-aware Embeddings

arXiv: https://arxiv.org/abs/1908.09357 · HTML: https://mlsanity.com/paper/1908.09357/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Aug 2019.

## Built on by (most cited first, 30 of 34)

- [Demonstration-Guided Reinforcement Learning with Learned Skills](https://mlsanity.com/paper/2107.10253/) (2021)
- [For SALE: State-Action Representation Learning for Deep Reinforcement Learning](https://mlsanity.com/paper/2306.02451/) (2023)
- [Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives](https://mlsanity.com/paper/2110.15360/) (2021)
- [PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Play](https://mlsanity.com/paper/2312.04549/) (2023)
- [HRP: Human Affordances for Robotic Pre-Training](https://mlsanity.com/paper/2407.18911/) (2024)
- [DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Control](https://mlsanity.com/paper/2409.12192/) (2024)
- [An Unbiased Look at Datasets for Visuo-Motor Pre-Training](https://mlsanity.com/paper/2310.09289/) (2023)
- [Hierarchical Skills for Efficient Exploration](https://mlsanity.com/paper/2110.10809/) (2021)
- [HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation](https://mlsanity.com/paper/2109.05490/) (2021)
- [LASER: Learning a Latent Action Space for Efficient Reinforcement Learning](https://mlsanity.com/paper/2103.15793/) (2021)
- [Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitation](https://mlsanity.com/paper/2305.16985/) (2023)
- [JueWu-MC: Playing Minecraft with Sample-efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2112.04907/) (2021)
- [Continuous MDP Homomorphisms and Homomorphic Policy Gradient](https://mlsanity.com/paper/2209.07364/) (2022)
- [Cross-Domain Policy Adaptation by Capturing Representation Mismatch](https://mlsanity.com/paper/2405.15369/) (2024)
- [Unsupervised Representation Learning in Deep Reinforcement Learning: A Review](https://mlsanity.com/paper/2208.14226/) (2022)
- [Hierarchical Neural Dynamic Policies](https://mlsanity.com/paper/2107.05627/) (2021)
- [Decoupled Exploration and Exploitation Policies for Sample-Efficient Reinforcement Learning](https://mlsanity.com/paper/2101.09458/) (2021)
- [One After Another: Learning Incremental Skills for a Changing World](https://mlsanity.com/paper/2203.11176/) (2022)
- [Policy Gradient Methods in the Presence of Symmetries and State Abstractions](https://mlsanity.com/paper/2305.05666/) (2023)
- [Learning minimal representations of stochastic processes with variational autoencoders](https://mlsanity.com/paper/2307.11608/) (2023)
- [SkillS: Adaptive Skill Sequencing for Efficient Temporally-Extended Exploration](https://mlsanity.com/paper/2211.13743/) (2022)
- [Leveraging Jumpy Models for Planning and Fast Learning in Robotic Domains](https://mlsanity.com/paper/2302.12617/) (2023)
- [Physical Reasoning Using Dynamics-Aware Models](https://mlsanity.com/paper/2102.10336/) (2021)
- [Dynamic Neighborhood Construction for Structured Large Discrete Action Spaces](https://mlsanity.com/paper/2305.19891/) (2023)
- [Leveraging Demonstrations with Latent Space Priors](https://mlsanity.com/paper/2210.14685/) (2022)
- [Reinforcement-Learning Portfolio Allocation with Dynamic Embedding of Market Information](https://mlsanity.com/paper/2501.17992/) (2025)
- [A Self-Supervised Auxiliary Loss for Deep RL in Partially Observable Settings](https://mlsanity.com/paper/2104.08492/) (2021)
- [Learning Routines for Effective Off-Policy Reinforcement Learning](https://mlsanity.com/paper/2106.02943/) (2021)
- [Learning Embeddings for Sequential Tasks Using Population of Agents](https://mlsanity.com/paper/2306.03311/) (2023)
- [DCT: Dual Channel Training of Action Embeddings for Reinforcement Learning with Large Discrete Action Spaces](https://mlsanity.com/paper/2306.15913/) (2023)

## Abstract (arXiv)

In this paper we consider self-supervised representation learning to improve sample efficiency in reinforcement learning (RL). We propose a forward prediction objective for simultaneously learning embeddings of states and action sequences. These embeddings capture the structure of the environment's dynamics, enabling efficient policy learning. We demonstrate that our action embeddings alone improve the sample efficiency and peak performance of model-free RL on control from low-dimensional states. By combining state and action embeddings, we achieve efficient learning of high-quality policies on goal-conditioned continuous control from pixel observations in only 1-2 million environment steps.