# DCT: Dual Channel Training of Action Embeddings for Reinforcement Learning with Large Discrete Action Spaces

arXiv: https://arxiv.org/abs/2306.15913 · HTML: https://mlsanity.com/paper/2306.15913/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 27 references that are in the map.
- First posted on arXiv 28 Jun 2023.

## Builds on (most cited first, 14 of 14)

- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Reinforcement learning based recommender systems: A survey](https://mlsanity.com/paper/2101.06286/) (2021)
- [Deep Reinforcement Learning in Large Discrete Action Spaces](https://mlsanity.com/paper/1512.07679/) (2015)
- [A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem](https://mlsanity.com/paper/1706.10059/) (2017)
- [Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning](https://mlsanity.com/paper/1809.02121/) (2018)
- [Learning Action Representations for Reinforcement Learning](https://mlsanity.com/paper/1902.00183/) (2019)
- [Hybrid Actor-Critic Reinforcement Learning in Parameterized Action Space](https://mlsanity.com/paper/1903.01344/) (2019)
- [Dynamics-aware Embeddings](https://mlsanity.com/paper/1908.09357/) (2019)
- [The Natural Language of Actions](https://mlsanity.com/paper/1902.01119/) (2019)
- [Controlling Assistive Robots with Learned Latent Actions](https://mlsanity.com/paper/1909.09674/) (2019)
- [Generalization to New Actions in Reinforcement Learning](https://mlsanity.com/paper/2011.01928/) (2020)
- [Learning to Run challenge solutions: Adapting reinforcement learning methods for neuromusculoskeletal environments](https://mlsanity.com/paper/1804.00361/) (2018)
- [Action Redundancy in Reinforcement Learning](https://mlsanity.com/paper/2102.11329/) (2021)
- [Simple Emergent Action Representations from Multi-Task Policy Training](https://mlsanity.com/paper/2210.09566/) (2022)

## Abstract (arXiv)

The ability to learn robust policies while generalizing over large discrete action spaces is an open challenge for intelligent systems, especially in noisy environments that face the curse of dimensionality. In this paper, we present a novel framework to efficiently learn action embeddings that simultaneously allow us to reconstruct the original action as well as to predict the expected future state. We describe an encoder-decoder architecture for action embeddings with a dual channel loss that balances between action reconstruction and state prediction accuracy. We use the trained decoder in conjunction with a standard reinforcement learning algorithm that produces actions in the embedding space. Our architecture is able to outperform two competitive baselines in two diverse environments: a 2D maze environment with more than 4000 discrete noisy actions, and a product recommendation task that uses real-world e-commerce transaction data. Empirical results show that the model results in cleaner action embeddings, and the improved representations help learn better policies with earlier convergence.