# Learning to Factor Policies and Action-Value Functions: Factored Action Space Representations for Deep Reinforcement learning

arXiv: https://arxiv.org/abs/1705.07269 · HTML: https://mlsanity.com/paper/1705.07269/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 May 2017.

## Built on by (most cited first, 15 of 15)

- [Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare](https://mlsanity.com/paper/2305.01738/) (2023)
- [Continuous MDP Homomorphisms and Homomorphic Policy Gradient](https://mlsanity.com/paper/2209.07364/) (2022)
- [Solving Continuous Control via Q-learning](https://mlsanity.com/paper/2210.12566/) (2022)
- [Imitation Learning with Concurrent Actions in 3D Games](https://mlsanity.com/paper/1803.05402/) (2018)
- [On Robot Grasp Learning Using Equivariant Models](https://mlsanity.com/paper/2306.06489/) (2023)
- [Policy Gradient Methods in the Presence of Symmetries and State Abstractions](https://mlsanity.com/paper/2305.05666/) (2023)
- [Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution](https://mlsanity.com/paper/2404.04253/) (2024)
- [No Prior Mask: Eliminate Redundant Action for Deep Reinforcement Learning](https://mlsanity.com/paper/2312.06258/) (2023)
- [REValueD: Regularised Ensemble Value-Decomposition for Factorisable Markov Decision Processes](https://mlsanity.com/paper/2401.08850/) (2024)
- [Dynamic Neighborhood Construction for Structured Large Discrete Action Spaces](https://mlsanity.com/paper/2305.19891/) (2023)
- [CANDID DAC: Leveraging Coupled Action Dimensions with Importance Differences in DAC](https://mlsanity.com/paper/2407.05789/) (2024)
- [An Investigation of Offline Reinforcement Learning in Factorisable Action Spaces](https://mlsanity.com/paper/2411.11088/) (2024)
- [MAN: Multi-Action Networks Learning](https://mlsanity.com/paper/2209.09329/) (2022)
- [Reinforcement Learning with Large Action Spaces for Neural Machine Translation](https://mlsanity.com/paper/2210.03053/) (2022)
- [Generalizable Hierarchical Skill Learning via Object-Centric Representation](https://mlsanity.com/paper/2510.21121/) (2025)

## Abstract (arXiv)

Deep Reinforcement Learning (DRL) methods have performed well in an increasing numbering of high-dimensional visual decision making domains. Among all such visual decision making problems, those with discrete action spaces often tend to have underlying compositional structure in the said action space. Such action spaces often contain actions such as go left, go up as well as go diagonally up and left (which is a composition of the former two actions). The representations of control policies in such domains have traditionally been modeled without exploiting this inherent compositional structure in the action spaces. We propose a new learning paradigm, Factored Action space Representations (FAR) wherein we decompose a control policy learned using a Deep Reinforcement Learning Algorithm into independent components, analogous to decomposing a vector in terms of some orthogonal basis vectors. This architectural modification of the control policy representation allows the agent to learn about multiple actions simultaneously, while executing only one of them. We demonstrate that FAR yields considerable improvements on top of two DRL algorithms in Atari 2600: FARA3C outperforms A3C (Asynchronous Advantage Actor Critic) in 9 out of 14 tasks and FARAQL outperforms AQL (Asynchronous n-step Q-Learning) in 9 out of 13 tasks.