mlsanityData to 14 Oct 2026

Adaptive Finite-Budget Training for CVaR Risk-Aware Q-Learning

arXiv 2608.04305 ↗First posted 5 Aug 2026cs.LG q-fin.RMcited by 0

On the map: Deep reinforcement learning methods → Adversarial robustness and RL in finance

Abstract (arXiv)

Risk-aware Q-learning (RaQL) provides a model-free, two-timescale estimator for dynamic risk objectives, but its finite-budget behavior remains fragile: fixed inner-loop hyperparameters can produce unstable value estimates, persistent Bellman residuals, and inefficient sample reuse. This paper proposes an adaptive training controller for Conditional Value-at-Risk (CVaR) RaQL and evaluates it on a daily Bitcoin trading task. The controller preserves the original CVaR estimator and Bellman fixed point; instead, it redesigns the training procedure through six coordinated mechanisms: per-cell inner-step sizing, outer-rate-matched decay synchronization, a short early correction for the VaR-like inner variable, a coverage-first-then-greedy sample allocation rule, progressive suffix aggregation of mature inner estimates, and data-driven calibration of key scales from online-observable quantities. Across 20 random seeds and 856,000 inner-transition samples, the controller reduces the mean empirical CVaR Bellman residual by approximately 85% relative to the fixed-parameter baseline (MeanBEQ: 1.2202 to 0.1854; MeanBEV: 1.1624 to 0.0535) and maintains stability across CVaR levels, discount factors, and training budgets. On the chronological out-of-sample test set, the learned policy attains a Sharpe ratio of 0.9281 with a maximum drawdown of 6.46% after transaction costs. Although buy-and-hold yields a higher cumulative return (35.43% vs. 23.61%), the adaptive policy achieves far lower volatility (9.57% vs. 47.93%), drawdown, and CVaR loss. These results demonstrate that adaptive finite-budget training design, applied solely to the training procedure without altering the risk objective, can materially improve the reliability and risk-adjusted performance of risk-aware Q-learning in financial applications.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods. Builds on 14 of its 36 references that are in the map. First posted on arXiv 5 Aug 2026.

Builds on

14 of 36 references.

References · 14

  1. 2015Prioritized Experience Replaycited by 1,099
  2. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  3. 2018Is Q-learning Provably Efficient?cited by 487
  4. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  5. 2014Algorithms for CVaR Optimization in MDPscited by 99
  6. 2016Learning values across many orders of magnitudecited by 85
  7. 2021A Statistical Analysis of Polyak-Ruppert Averaged Q-learningcited by 29
  8. 2021Return-based Scaling: Yet Another Normalisation Trick for Deep RLcited by 23
  9. 2014Policy Gradients for CVaR-Constrained MDPscited by 21
  10. 2020Option Hedging with Risk Averse Reinforcement Learningcited by 13
  11. 2021Risk-averse autonomous systems: A brief history and recent developments from the perspective of optimal controlcited by 12
  12. 2018Stochastic Approximation for Risk-aware Markov Decision Processescited by 10
2 more
  1. 2020Risk-Sensitive Reinforcement Learning: a Martingale Approach to Reward Uncertaintycited by 5
  2. 2024Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximationcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reward Redistribution for CVaR MDPs using a Bellman Operator on L-infinity
  2. 2018The QLBS Q-Learner Goes NuQLear: Fitted Q Iteration, Inverse RL, and Option Portfolios
  3. 2026Long-Run Conditional Value-at-Risk Reinforcement Learning
  4. 2026Trading Confidence: Comprehensive Uncertainty Estimation in Algorithmic Trading
  5. 2026Sign-Separated Asymmetric Finite-Time Error Analysis of Q-Learning
  6. 2025Tail-Safe Hedging: Explainable Risk-Sensitive Reinforcement Learning with a White-Box CBF--QP Safety Layer in Arbitrage-Free Markets
  7. 2025Quantile Q-Learning: Revisiting Offline Extreme Q-Learning with Quantile Regression
  8. 2026Revisiting TD Target Aggregation under Uncertainty in Q-Learning