# Learning Action Representations for Reinforcement Learning

arXiv: https://arxiv.org/abs/1902.00183 · HTML: https://mlsanity.com/paper/1902.00183/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Feb 2019.

## Built on by (most cited first, 30 of 75)

- [Tutorial on amortized optimization](https://mlsanity.com/paper/2202.00665/) (2022)
- [For SALE: State-Action Representation Learning for Deep Reinforcement Learning](https://mlsanity.com/paper/2306.02451/) (2023)
- [Supervised Training of Conditional Monge Maps](https://mlsanity.com/paper/2206.14262/) (2022)
- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learning](https://mlsanity.com/paper/2306.13229/) (2023)
- [Off-Policy Evaluation for Large Action Spaces via Embeddings](https://mlsanity.com/paper/2202.06317/) (2022)
- [HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation](https://mlsanity.com/paper/2109.05490/) (2021)
- [Learning Large Neighborhood Search Policy for Integer Programming](https://mlsanity.com/paper/2111.03466/) (2021)
- [Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modeling](https://mlsanity.com/paper/2305.08062/) (2023)
- [LASER: Learning a Latent Action Space for Efficient Reinforcement Learning](https://mlsanity.com/paper/2103.15793/) (2021)
- [What Do Latent Action Models Actually Learn?](https://mlsanity.com/paper/2506.15691/) (2025)
- [Continuous MDP Homomorphisms and Homomorphic Policy Gradient](https://mlsanity.com/paper/2209.07364/) (2022)
- [Cross-Domain Policy Adaptation by Capturing Representation Mismatch](https://mlsanity.com/paper/2405.15369/) (2024)
- [PRISE: LLM-Style Sequence Compression for Learning Temporal Action Abstractions in Control](https://mlsanity.com/paper/2402.10450/) (2024)
- [Deep hierarchical reinforcement agents for automated penetration testing](https://mlsanity.com/paper/2109.06449/) (2021)
- [Unsupervised Representation Learning in Deep Reinforcement Learning: A Review](https://mlsanity.com/paper/2208.14226/) (2022)
- [Off-Policy Evaluation of Slate Bandit Policies via Optimizing Abstraction](https://mlsanity.com/paper/2402.02171/) (2024)
- [A Survey of State Representation Learning for Deep Reinforcement Learning](https://mlsanity.com/paper/2506.17518/) (2025)
- [Action Tokenizer Matters in In-Context Imitation Learning](https://mlsanity.com/paper/2503.01206/) (2025)
- [Generative Slate Recommendation with Reinforcement Learning](https://mlsanity.com/paper/2301.08632/) (2023)
- [POTEC: Off-Policy Learning for Large Action Spaces via Two-Stage Policy Decomposition](https://mlsanity.com/paper/2402.06151/) (2024)
- [Learn the Time to Learn: Replay Scheduling in Continual Learning](https://mlsanity.com/paper/2209.08660/) (2022)
- [Uncertainty Estimation Using Riemannian Model Dynamics for Offline Reinforcement Learning](https://mlsanity.com/paper/2102.11327/) (2021)
- [Perceiving the World: Question-guided Reinforcement Learning for Text-based Games](https://mlsanity.com/paper/2204.09597/) (2022)
- [ResAct: Reinforcing Long-term Engagement in Sequential Recommendation with Residual Actor](https://mlsanity.com/paper/2206.02620/) (2022)
- [AD3: Implicit Action is the Key for World Models to Distinguish the Diverse Visual Distractors](https://mlsanity.com/paper/2403.09976/) (2024)
- [Action Redundancy in Reinforcement Learning](https://mlsanity.com/paper/2102.11329/) (2021)
- [Reinforcement Learning in Presence of Discrete Markovian Context Evolution](https://mlsanity.com/paper/2202.06557/) (2022)
- [Policy Gradient Methods in the Presence of Symmetries and State Abstractions](https://mlsanity.com/paper/2305.05666/) (2023)
- [Solving Large-Scale Extensive-Form Network Security Games via Neural Fictitious Self-Play](https://mlsanity.com/paper/2106.00897/) (2021)

## Abstract (arXiv)

Most model-free reinforcement learning methods leverage state representations (embeddings) for generalization, but either ignore structure in the space of actions or assume the structure is provided a priori. We show how a policy can be decomposed into a component that acts in a low-dimensional space of action representations and a component that transforms these representations into actual actions. These representations improve generalization over large, finite action sets by allowing the agent to infer the outcomes of actions similar to actions already taken. We provide an algorithm to both learn and use action representations and provide conditions for its convergence. The efficacy of the proposed method is demonstrated on large-scale real-world problems.