On Gaussian approximation for entropy-regularized Q-learning with function approximation
Not on the map. Why?
Abstract (arXiv)
In this paper, we derive rates of convergence in the high-dimensional central limit theorem for Polyak--Ruppert averaged iterates generated by entropy-regularized asynchronous Q-learning with linear function approximation and a polynomial stepsize k-ω, ω ∈ (1/2,1). Assuming that the sequence of observed triples (sk,ak,sk+1)k ≥ 0 forms a uniformly geometrically ergodic Markov chain, and under suitable regularity conditions for the projected soft Bellman equation, we establish a Gaussian approximation bound in the convex distance with rate of order n-1/4, up to polylogarithmic factors in n, where n is the number of samples used by the algorithm. To obtain this result, we combine a linearization of the soft Bellman recursion with a Gaussian approximation for the leading martingale term. Finally, we derive high-order moment bounds for the algorithm's last iterate, which might be of independent interest.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 41 references that are in the map. First posted on arXiv 17 May 2026.
Builds on
20 of 41 references.
References · 20
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
- 2018Is Q-learning Provably Efficient?cited by 487
- 2020Finite-Time Analysis of Asynchronous Stochastic Approximation and Q-Learningcited by 96
- 2021Is Q-Learning Minimax Optimal? A Tight Sample Complexity Analysiscited by 92
- 2019Finite-Sample Analysis for SARSA with Linear Function Approximationcited by 85
- 2019Variance-reduced Q-learning is minimax optimalcited by 73
- 2019Finite-Sample Analysis of Nonlinear Stochastic Approximation with Applications in Reinforcement Learningcited by 67
- 2020Finite Time Analysis of Linear Two-timescale Stochastic Approximation with Markovian Noisecited by 53
- 2022Finite-time High-probability Bounds for Polyak-Ruppert Averaged Iterates of Linear Stochastic Approximationcited by 41
- 2021A Statistical Analysis of Polyak-Ruppert Averaged Q-learningcited by 29
- 2021Tight High Probability Bounds for Linear Stochastic Approximation with Fixed Stepsizecited by 27
8 more
- 2024Gaussian Approximation and Multiplier Bootstrap for Polyak-Ruppert Averaged Linear Stochastic Approximation with Applications to TD Learningcited by 19
- 2024Statistical Inference for Policy Evaluation with Temporal Difference Learningcited by 15
- 2025Statistical inference for Linear Stochastic Approximation with Markovian Noisecited by 13
- 2025Uncertainty quantification for Markov chain induced martingales with application to temporal difference learningcited by 11
- 2025Gaussian Approximation and Multiplier Bootstrap for Stochastic Gradient Descentcited by 10
- 2025Gaussian Approximation for Two-Timescale Linear Stochastic Approximationcited by 5
- 2025Central Limit Theorems for Asynchronous Averaged Q-Learningcited by 4
- 2026Gaussian Approximation for Asynchronous Q-learningcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Q-Learning with Stable Infinite-Dimensional Linear Function Approximation
- 2024Regularized Q-Learning with Linear Function Approximation
- 2025A General-Purpose Theorem for High-Probability Bounds of Stochastic Approximation with Polyak Averaging
- 2023Soft Quantization using Entropic Regularization
- 2024Constant Stepsize Q-learning: Distributional Convergence, Bias and Extrapolation
- 2026Optimistic Training and Convergence of Q-Learning -- Extended Version
- 2024Optimal Rates of Convergence for Entropy Regularization in Discounted Markov Decision Processes
- 2026Achieving varepsilon-2 Dependence for Average-Reward Q-Learning with a New Contraction Principle