# MAN: Multi-Action Networks Learning

arXiv: https://arxiv.org/abs/2209.09329 · HTML: https://mlsanity.com/paper/2209.09329/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 19 Sep 2022.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [Deep Recurrent Q-Learning for Partially Observable MDPs](https://mlsanity.com/paper/1507.06527/) (2015)
- [Implicit Quantile Networks for Distributional Reinforcement Learning](https://mlsanity.com/paper/1806.06923/) (2018)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [Equivalence Between Policy Gradients and Soft Q-Learning](https://mlsanity.com/paper/1704.06440/) (2017)
- [Deep Reinforcement Learning in Large Discrete Action Spaces](https://mlsanity.com/paper/1512.07679/) (2015)
- [Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning](https://mlsanity.com/paper/1611.01929/) (2016)
- [Action Branching Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1711.08946/) (2017)
- [Learning values across many orders of magnitude](https://mlsanity.com/paper/1602.07714/) (2016)
- [Massively Parallel Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1507.04296/) (2015)
- [Discrete Sequential Prediction of Continuous Actions for Deep RL](https://mlsanity.com/paper/1705.05035/) (2017)
- [Observe and Look Further: Achieving Consistent Performance on Atari](https://mlsanity.com/paper/1805.11593/) (2018)
- [Accelerated Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1803.02811/) (2018)
- [Efficient Parallel Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1705.04862/) (2017)
- [Q-Learning in enormous action spaces via amortized approximate maximization](https://mlsanity.com/paper/2001.08116/) (2020)
- [Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tightening](https://mlsanity.com/paper/1611.01606/) (2016)
- [Learning to Factor Policies and Action-Value Functions: Factored Action Space Representations for Deep Reinforcement learning](https://mlsanity.com/paper/1705.07269/) (2017)
- [Reinforcement Learning in Factored Action Spaces using Tensor Decompositions](https://mlsanity.com/paper/2110.14538/) (2021)
- [Deep Quality-Value (DQV) Learning](https://mlsanity.com/paper/1810.00368/) (2018)

## Built on by (most cited first, 2 of 2)

- [OpenVR: Teleoperation for Manipulation](https://mlsanity.com/paper/2305.09765/) (2023)
- [Self-Driving Car Racing: Application of Deep Reinforcement Learning](https://mlsanity.com/paper/2410.22766/) (2024)

## Abstract (arXiv)

Learning control policies with large discrete action spaces is a challenging problem in the field of reinforcement learning due to present inefficiencies in exploration. With high dimensional action spaces, there are a large number of potential actions in each individual dimension over which policies would be learned. In this work, we introduce a Deep Reinforcement Learning (DRL) algorithm call Multi-Action Networks (MAN) Learning that addresses the challenge of high-dimensional large discrete action spaces. We propose factorizing the N-dimension action space into N 1-dimensional components, known as sub-actions, creating a Value Neural Network for each sub-action. Then, MAN uses temporal-difference learning to train the networks synchronously, which is simpler than training a single network with a large action output directly. To evaluate the proposed method, we test MAN on three scenarios: an n-dimension maze task, a block stacking task, and then extend MAN to handle 12 games from the Atari Arcade Learning environment with 18 action spaces. Our results indicate that MAN learns faster than both Deep Q-Learning and Double Deep Q-Learning, implying our method is a better performing synchronous temporal difference algorithm than those currently available for large discrete action spaces.