mlsanityData to 7 Oct 2026

Q-Learning in enormous action spaces via amortized approximate maximization

arXiv 2001.08116 ↗First posted 22 Jan 2020cs.LG cs.AI stat.MLcited by 30

Not on the map. Why?

Abstract (arXiv)

Applying Q-learning to high-dimensional or continuous action spaces can be difficult due to the required maximization over the set of possible actions. Motivated by techniques from amortized inference, we replace the expensive maximization over all actions with a maximization over a small subset of possible actions sampled from a learned proposal distribution. The resulting approach, which we dub Amortized Q-learning (AQL), is able to handle discrete, continuous, or hybrid action spaces while maintaining the benefits of Q-learning. Our experiments on continuous control tasks with up to 21 dimensional actions show that AQL outperforms D3PG (Barth-Maron et al, 2018) and QT-Opt (Kalashnikov et al, 2018). Experiments on structured discrete action spaces demonstrate that AQL can efficiently learn good policies in spaces with thousands of discrete actions.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 5; 2023: 5; 2024: 9; 2025: 1; 2026: 1.

30 papers.

Sort:

By area:

  1. 2022Tutorial on amortized optimizationcited by 83
  2. 2023Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcarecited by 41
  3. 2021No-Press Diplomacy from Scratchcited by 31
  4. 2021NeurWIN: Neural Whittle Index Network For Restless Bandits Via Deep RLcited by 27
  5. 2021Reinforced Few-Shot Acquisition Function Learning for Bayesian Optimizationcited by 22
  6. 2022Solving Continuous Control via Q-learningcited by 21
  7. 2021Deep hierarchical reinforcement agents for automated penetration testingcited by 19
  8. 2022ALMA: Hierarchical Learning for Composite Multi-Agent Taskscited by 17
  9. 2021Centralizing State-Values in Dueling Networks for Multi-Robot Reinforcement Learning Mapless Navigationcited by 13
  10. 2021Adjacency constraint for efficient hierarchical reinforcement learningcited by 10
  11. 2021Reversible Action Design for Combinatorial Optimization with Reinforcement Learningcited by 7
  12. 2023Graph Reinforcement Learning for Network Control via Bi-Level Optimizationcited by 7
18 more
  1. 2023Weakly Coupled Deep Q-Networkscited by 6
  2. 2024REValueD: Regularised Ensemble Value-Decomposition for Factorisable Markov Decision Processescited by 6
  3. 2024Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolutioncited by 6
  4. 2023An Invitation to Deep Reinforcement Learningcited by 5
  5. 2024Stochastic Q-learning for Large Discrete Action Spacescited by 5
  6. 2024How to Choose a Reinforcement-Learning Algorithmcited by 3
  7. 2024BraVE: Offline Reinforcement Learning for Discrete Combinatorial Action Spacescited by 3
  8. 2024An Investigation of Offline Reinforcement Learning in Factorisable Action Spacescited by 3
  9. 2022MAN: Multi-Action Networks Learningcited by 2
  10. 2021Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific researchcited by 1
  11. 2023Online Network Source Optimization with Graph-Kernel MABcited by 1
  12. 2021Towards Autonomous Satellite Communications: An AI-based Framework to Address System-level Challengescited by 0
  13. 2022SlateFree: a Model-Free Decomposition for Reinforcement Learning with Slate Actionscited by 0
  14. 2024A comparison of RL-based and PID controllers for 6-DOF swimming robots: hybrid underwater object trackingcited by 0
  15. 2024How to discretize continuous state-action spaces in Q-learning: A symbolic control approachcited by 0
  16. 2024Learning in complex action spaces without policy gradientscited by 0
  17. 2025SAINT: Attention-Based Policies for Discrete Combinatorial Action Spacescited by 0
  18. 2026Improving and Accelerating Offline RL in Large Discrete Action Spaces with Structured Policy Initializationcited by 0