# The Natural Language of Actions

arXiv: https://arxiv.org/abs/1902.01119 · HTML: https://mlsanity.com/paper/1902.01119/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Feb 2019.

## Built on by (most cited first, 28 of 28)

- [Tutorial on amortized optimization](https://mlsanity.com/paper/2202.00665/) (2022)
- [For SALE: State-Action Representation Learning for Deep Reinforcement Learning](https://mlsanity.com/paper/2306.02451/) (2023)
- [Supervised Training of Conditional Monge Maps](https://mlsanity.com/paper/2206.14262/) (2022)
- [Off-Policy Evaluation for Large Action Spaces via Embeddings](https://mlsanity.com/paper/2202.06317/) (2022)
- [On Entropy Control in LLM-RL Algorithms](https://mlsanity.com/paper/2509.03493/) (2025)
- [Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modeling](https://mlsanity.com/paper/2305.08062/) (2023)
- [Continuous MDP Homomorphisms and Homomorphic Policy Gradient](https://mlsanity.com/paper/2209.07364/) (2022)
- [Uncertainty Estimation Using Riemannian Model Dynamics for Offline Reinforcement Learning](https://mlsanity.com/paper/2102.11327/) (2021)
- [Generative Planning for Temporally Coordinated Exploration in Reinforcement Learning](https://mlsanity.com/paper/2201.09765/) (2022)
- [Action Redundancy in Reinforcement Learning](https://mlsanity.com/paper/2102.11329/) (2021)
- [Policy Gradient Methods in the Presence of Symmetries and State Abstractions](https://mlsanity.com/paper/2305.05666/) (2023)
- [Embedding-Aligned Language Models](https://mlsanity.com/paper/2406.00024/) (2024)
- [Learning Action Embeddings for Off-Policy Evaluation](https://mlsanity.com/paper/2305.03954/) (2023)
- [Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spaces](https://mlsanity.com/paper/2509.22963/) (2025)
- [No Prior Mask: Eliminate Redundant Action for Deep Reinforcement Learning](https://mlsanity.com/paper/2312.06258/) (2023)
- [Stochastic Q-learning for Large Discrete Action Spaces](https://mlsanity.com/paper/2405.10310/) (2024)
- [Dynamic Neighborhood Construction for Structured Large Discrete Action Spaces](https://mlsanity.com/paper/2305.19891/) (2023)
- [Representation-Driven Reinforcement Learning](https://mlsanity.com/paper/2305.19922/) (2023)
- [Learning Actionable World Models for Industrial Process Control](https://mlsanity.com/paper/2503.01411/) (2025)
- [Locality Matters: A Scalable Value Decomposition Approach for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2109.10632/) (2021)
- [Learning Natural Language Generation from Scratch](https://mlsanity.com/paper/2109.09371/) (2021)
- [Reinforcement Learning with Large Action Spaces for Neural Machine Translation](https://mlsanity.com/paper/2210.03053/) (2022)
- [Diverse Policy Optimization for Structured Action Space](https://mlsanity.com/paper/2302.11917/) (2023)
- [Learning Routines for Effective Off-Policy Reinforcement Learning](https://mlsanity.com/paper/2106.02943/) (2021)
- [More Efficient Exploration with Symbolic Priors on Action Sequence Equivalences](https://mlsanity.com/paper/2110.10632/) (2021)
- [DCT: Dual Channel Training of Action Embeddings for Reinforcement Learning with Large Discrete Action Spaces](https://mlsanity.com/paper/2306.15913/) (2023)
- [LOD: Latent Objective Discovery in Heterogeneous Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2505.09756/) (2025)
- [LLM-Guided Contextual Action Evaluation for Operational Decisions in Industrial Processes](https://mlsanity.com/paper/2608.24156/) (2026)

## Abstract (arXiv)

We introduce Act2Vec, a general framework for learning context-based action representation for Reinforcement Learning. Representing actions in a vector space help reinforcement learning algorithms achieve better performance by grouping similar actions and utilizing relations between different actions. We show how prior knowledge of an environment can be extracted from demonstrations and injected into action vector representations that encode natural compatible behavior. We then use these for augmenting state representations as well as improving function approximation of Q-values. We visualize and test action embeddings in three domains including a drawing task, a high dimensional navigation task, and the large action space domain of StarCraft II.