Stability and Generalization for Bellman Residuals
On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF
also in 2 more
Abstract (arXiv)
Offline reinforcement learning and offline inverse reinforcement learning aim to recover near-optimal value functions or reward models from a fixed batch of logged trajectories, yet current practice still struggles to enforce Bellman consistency. Bellman residual minimization (BRM) has emerged as an attractive remedy, as a globally convergent stochastic gradient descent-ascent based method for BRM has been recently discovered. However, its statistical behavior in the offline setting remains largely unexplored. In this paper, we close this statistical gap. Our analysis introduces a single Lyapunov potential that couples SGDA runs on neighbouring datasets and yields an O(1/n) on-average argument-stability bound-doubling the best known sample-complexity exponent for convex-concave saddle problems. The same stability constant translates into the O(1/n) excess risk bound for BRM, without variance reduction, extra regularization, or restrictive independence assumptions on minibatch sampling. The results hold for standard neural-network parameterizations and minibatch SGD.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 15 of its 28 references that are in the map. First posted on arXiv 26 Aug 2025.
Builds on
15 of 28 references.
References · 15
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
- 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
- 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
- 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
- 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
- 2018Deep Reinforcement Learning and the Deadly Triadcited by 174
- 2023Handbook of Convergence Theorems for (Stochastic) Gradient Methodscited by 163
- 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 116
- 2020Global Convergence and Variance-Reduced Optimization for a Class of Nonconvex-Nonconcave Minimax Problemscited by 38
- 2022Target Network and Truncation Overcome The Deadly Triad in Q-Learningcited by 31
- 2021A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learningcited by 26
3 more
- 2022Stability and Generalization for Markov Chain Stochastic Gradient Methodscited by 14
- 2022Structural Estimation of Markov Decision Processes in High-Dimensional State Space with Finite-Time Guaranteescited by 12
- 2023Error bounds, PL condition, and quadratic growth for weakly convex functions, and linear convergences of proximal point methodscited by 7
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Lifted Bellman Linear Programming for Offline Reinforcement Learning
- 2026Residuals-based Offline Reinforcement Learning
- 2025Bellman Calibration for V-Learning in Offline Reinforcement Learning
- 2022Bellman Residual Orthogonalization for Offline Reinforcement Learning
- 2026Offline Policy Evaluation via Mixed Bellman Residuals and Adaptive Critic Representations
- 2024The Role of Inherent Bellman Error in Offline Reinforcement Learning with Linear Function Approximation
- 2026Deep Weighted Bellman Residual Minimization for Q^* Estimation
- 2026Markovian Nonconvex ADMM for Reinforcement Learning: Bellman-Resolvent Stability Beyond Smooth Blocks