mlsanityData to 14 Oct 2026

Increasing the Action Gap: New Operators for Reinforcement Learning

arXiv 1512.04860 ↗First posted 15 Dec 2015cs.AI cs.LGcited by 49

Not on the map. Why?

Abstract (arXiv)

This paper introduces new optimality-preserving operators on Q-functions. We first describe an operator for tabular representations, the consistent Bellman operator, which incorporates a notion of local policy consistency. We show that this local consistency leads to an increase in the action gap at each state; increasing this gap, we argue, mitigates the undesirable effects of approximation and estimation errors on the induced greedy policies. This operator can also be applied to discretized continuous space and time problems, and we provide empirical results evidencing superior performance in this context. Extending the idea of a locally consistent operator, we then derive sufficient conditions for an operator to preserve optimality, leading to a family of operators which includes our consistent Bellman operator. As corollaries we provide a proof of optimality for Baird's advantage learning algorithm and derive other gap-increasing operators with interesting properties. We conclude with an empirical study on 60 Atari 2600 games illustrating the strong potential of these new operators.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). First posted on arXiv 15 Dec 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 10; 2022: 17; 2023: 7; 2024: 4; 2025: 4; 2026: 6.

49 papers.

Sort:

By area:

  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  2. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 89
  3. 2021Offline Reinforcement Learning as Anti-Explorationcited by 44
  4. 2022A Modern Self-Referential Weight Matrix That Learns to Modify Itselfcited by 41
  5. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 36
  6. 2023Multi-agent Reinforcement Learning: A Comprehensive Surveycited by 33
  7. 2022The Phenomenon of Policy Churncited by 28
  8. 2021Deciding What to Learn: A Rate-Distortion Approachcited by 23
  9. 2022A Ranking Game for Imitation Learningcited by 18
  10. 2021The Value of Information When Deciding What to Learncited by 15
  11. 2022Distributional Hamilton-Jacobi-Bellman Equations for Continuous-Time Reinforcement Learningcited by 12
  12. 2023Accelerating Value Iteration with Anchoringcited by 11
37 more
  1. 2023VA-learning as a more efficient alternative to Q-learningcited by 11
  2. 2021Implicitly Regularized RL with Implicit Q-Valuescited by 9
  3. 2024Nonparametric Bellman Mappings for Reinforcement Learning: Application to Robust Adaptive Filteringcited by 8
  4. 2022Lazy-MDPs: Towards Interpretable Reinforcement Learning by Learning When to Actcited by 7
  5. 2022Designing Rewards for Fast Learningcited by 7
  6. 2022Dynamic selection of p-norm in linear adaptive filtering via online kernel-based reinforcement learningcited by 7
  7. 2024Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learningcited by 7
  8. 2024Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PCcited by 7
  9. 2021ShinRL: A Library for Evaluating RL Algorithms from Theoretical and Practical Perspectivescited by 5
  10. 2022Planning to the Information Horizon of BAMDPs via Epistemic State Abstractioncited by 4
  11. 2023Bayesian Reinforcement Learning with Limited Cognitive Loadcited by 4
  12. 2023Regularization and Variance-Weighted Regression Achieves Minimax Optimality in Linear MDPs: Theory and Practicecited by 4
  13. 2023Proximal Bellman mappings for reinforcement learning and their application to robust adaptive filteringcited by 4
  14. 2021Deep Reinforcement Learning for Demand Driven Services in Logistics and Transportation Systems: A Surveycited by 3
  15. 2023Parameterized Projected Bellman Operatorcited by 3
  16. 2021Bad-Policy Density: A Measure of Reinforcement Learning Hardnesscited by 2
  17. 2022Simultaneously Updating All Persistence Values in Reinforcement Learningcited by 2
  18. 2025RL-DAUNCE: Reinforcement Learning-Driven Data Assimilation with Uncertainty-Aware Constrained Ensemblescited by 2
  19. 2021Geometric Value Iteration: Dynamic Error-Aware KL Regularization for Reinforcement Learningcited by 1
  20. 2022Smoothing Advantage Learningcited by 1
  21. 2022Robust Action Gap Increasing with Clipped Advantage Learningcited by 1
  22. 2024Topological Foundations of Reinforcement Learningcited by 1
  23. 2025Mirror Descent Actor Critic via Bounded Advantage Learningcited by 1
  24. 2025Bellman operator convergence enhancements in reinforcement learning algorithmscited by 1
  25. 2026AdamO: A Collapse-Suppressed Optimizer for Offline RLcited by 1
  26. 2026The Sim-to-Real Gap of Foundation Model Agents: A Unified MDP Perspectivecited by 1
  27. 2021Utilizing Skipped Frames in Action Repeats via Pseudo-Actionscited by 0
  28. 2022Simultaneous Double Q-learning with Conservative Advantage Learning for Actor-Critic Methodscited by 0
  29. 2022Enforcing KL Regularization in General Tsallis Entropy Reinforcement Learning via Advantage Learningcited by 0
  30. 2022Set-based value operators for non-stationary Markovian environmentscited by 0
  31. 2022Maximum Correntropy Value Decomposition for Multi-agent Deep Reinforcemen Learningcited by 0
  32. 2022online and lightweight kernel-based approximated policy iteration for dynamic p-norm linear adaptive filteringcited by 0
  33. 2025Average Reward Reinforcement Learning for Omega-Regular and Mean-Payoff Objectivescited by 0
  34. 2026Towards Model-Free Learning in Dynamic Population Games: An Application to Karma Economiescited by 0
  35. 2026Accelerating Q-learning through Efficient Value-Sharing across Actionscited by 0
  36. 2026An Irreducible Quantum Advantage in Aligning World Models with Realitycited by 0
  37. 2026A Convergence Framework for Deep V-Learning: Error Propagation and Sharp Action-Gap Boundscited by 0