mlsanityData to 14 Oct 2026

Convergence of regularized agent-state-based Q-learning in POMDPs

arXiv 2508.21314 ↗First posted 29 Aug 2025cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

In this paper, we present a framework to understand the convergence of commonly used Q-learning reinforcement learning algorithms in practice. Two salient features of such algorithms are: (i)~the Q-table is recursively updated using an agent state (such as the state of a recurrent neural network) which is not a belief state or an information state and (ii)~policy regularization is often used to encourage exploration and stabilize the learning algorithm. We investigate the simplest form of such Q-learning algorithms which we call regularized agent-state-based Q-learning (RASQL) and show that it converges under mild technical conditions to the fixed point of an appropriately defined regularized MDP, which depends on the stationary distribution induced by the behavioral policy. We also show that a similar analysis continues to work for a variant of RASQL that learns periodic policies. We present numerical examples to illustrate that the empirical convergence behavior matches with the proposed theoretical limit.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 27 of its 42 references that are in the map. First posted on arXiv 29 Aug 2025.

Builds on

27 of 42 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Agent-state based policies in POMDPs: Beyond belief-state MDPs24 Sep
  2. 2024Periodic agent-state based Q-learning for POMDPs8 Jul

Other references · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  5. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  6. 2017Deep Reinforcement Learning that Matterscited by 754
  7. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  8. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
  9. 2019A Theory of Regularized Markov Decision Processescited by 286
  10. 2017A unified view of entropy-regularized Markov decision processescited by 182
  11. 2018Understanding the impact of entropy on policy optimizationcited by 176
  12. 2018Deep Variational Reinforcement Learning for POMDPscited by 125
13 more
  1. 2016DESPOT: Online POMDP Planning with Regularizationcited by 106
  2. 2020Approximate information state for approximate planning and reinforcement learning in partially observed systemscited by 70
  3. 2018Differentiable Dynamic Programming for Structured Prediction and Attentioncited by 58
  4. 2021Memory-based Deep Reinforcement Learning for POMDPscited by 54
  5. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 54
  6. 2018On Improving Deep Reinforcement Learning for POMDPscited by 50
  7. 2021Simple Agent, Complex Environment: Efficient Reinforcement Learning with Agent Statescited by 43
  8. 2021Convergence of Finite Memory Q-Learning for POMDPs and Near Optimality of Learned Policies under Filter Stabilitycited by 22
  9. 2022Reinforcement Learning in Non-Markovian Environmentscited by 10
  10. 2023Q-Learning for Stochastic Control under General Information Structures and Non-Markovian Environmentscited by 7
  11. 2021Smoother Entropy for Active State Trajectory Estimation and Obfuscation in POMDPscited by 6
  12. 2024The Limits of Pure Exploration in POMDPs: When the Observation Entropy is Enoughcited by 6
  13. 2024Entropy-regularized Point-based Value Iterationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Periodic Regularized Q-Learning
  2. 2024On Convergence of Average-Reward Q-Learning in Weakly Communicating Markov Decision Processes
  3. 2026Mixing Regularization for Average-Reward Q-Learning and Q-Planning
  4. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularization
  5. 2023A Connection between One-Step Regularization and Critic Regularization in Reinforcement Learning
  6. 2026Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities
  7. 2021Q-Learning for MDPs with General Spaces: Convergence and Near Optimality via Quantization under Weak Continuity
  8. 2026Policy Gradient Methods for Non-Markovian Reinforcement Learning