Nonparametric Variance-Penalized Actor-Critic: Statistical Inference for Risk-Sensitive Reinforcement Learning

arXiv 2609.14327 ↗First posted 13 Sep 2026cs.LG stat.MLcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Variance penalization is a principled approach to risk-sensitive reinforcement learning (RL) that explicitly trades expected return for policy stability. Existing methods require a dedicated second critic to estimate return variance online, adding architectural complexity and compounding estimation error during learning. We propose a nonparametric variance-penalized actor-critic (VPAC) framework that replaces the variance critic with statistically grounded online estimators based on bootstrapping and random scaling, techniques drawn from the statistical inference literature for stochastic approximation. These estimators require no auxiliary network, maintain a single-critic architecture, and produce variance penalties that are bounded by construction, enabling clean convergence analysis. We establish almost-sure convergence for both a variance-penalized Q-learning algorithm and a two-timescale actor-critic variant via the ordinary differential equation (ODE) method, requiring only that variance estimates remain bounded rather than consistent. Empirically, we evaluate across discrete and continuous stochastic environments, demonstrating that the proposed methods match or exceed the variance reduction achieved by the existing dual-critic VPAC baseline while eliminating the overhead of a second critic. We further validate on a high-temperature superconductor (HTS) manufacturing case study, where VPAC-RS (Random Scaling) achieves a 74% reduction in steady-state critical current variability and a 63% reduction in episode return standard deviation, translating directly to improved yield consistency. Our results establish nonparametric statistical inference as a practical and theoretically sound alternative to auxiliary critics for risk-sensitive RL.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 19 of its 48 references that are in the map. First posted on arXiv 13 Sep 2026.

Builds on

19 of 48 references.

Closest ancestors · 2

Same direction, newest first

Other references · 17

2015End-to-End Training of Deep Visuomotor Policiescited by 875
2017A Distributional Perspective on Reinforcement Learningcited by 765
2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
2013Risk-sensitive Reinforcement Learningcited by 130
2020A Finite Time Analysis of Two Time-Scale Actor Critic Methodscited by 122
2014Optimizing the CVaR via Samplingcited by 99
2016Statistical Inference for Model Parameters in Stochastic Gradient Descentcited by 71
2019Risk-Averse Trust Region Optimization for Reward-Volatility Reductioncited by 54
2014Variance-Constrained Actor-Critic Algorithms for Discounted and Average Reward MDPscited by 28
2021Online Bootstrap Inference For Policy Evaluation in Reinforcement Learningcited by 26

5 more

2023Online Statistical Inference for Nonlinear Stochastic Approximation with Markovian Datacited by 20
2019On Constructing Confidence Region for Model Parameters in Stochastic Gradient Descent via Batch Meanscited by 13
2013Variance Adjusted Actor Critic Algorithmscited by 11
2021Variance Penalized On-Policy and Off-Policy Actor-Criticcited by 10
2020Risk-Sensitive Deep RL: Variance-Constrained Actor-Critic Provably Finds Globally Optimal Policycited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way