Optimal Value Inference for Reinforcement Learning
On the map: Deep reinforcement learning methods → RL theory and sample complexity
Abstract (arXiv)
We study offline inference for the optimal value in reinforcement learning under finite state and action spaces. Two new nuisances are derived as fixed points of a self-induced Bellman equation, in which we approximate the maximum Bellman operator by its softmax correspondence. We propose a debiased estimator through the Neyman orthogonality and establish its asymptotic normality under diverging horizons even when the behavior policy changes with time, as long as the nuisances have the statistical rates that can be achieved by many machine learning methods. We provide a concrete estimating procedure for these nuisances and show they can lead to valid inference. Synthetic experiments validate the numerical performance of our inference method, and we implement it in real-life decision-making problems, including bike repositioning and AI agentic tool use.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 5 of its 64 references that are in the map. First posted on arXiv 9 Sep 2026.
Builds on
5 of 64 references.
References · 5
- 2020Reinforcement Learning for Non-Stationary Markov Decision Processes: The Blessing of (More) Optimismcited by 76
- 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiencycited by 56
- 2020Model-Free Non-Stationary RL: Near-Optimal Regret and Applications in Multi-Agent RL and Inventory Controlcited by 17
- 2025Inference on Optimal Policy Values and Other Irregular Functionals via Softmax Smoothingcited by 3
- 2025Semiparametric Off-Policy Inference for Optimal Policy Values under Possible Non-Uniquenesscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2022Bellman Residual Orthogonalization for Offline Reinforcement Learning
- 2025Bellman Calibration for V-Learning in Offline Reinforcement Learning
- 2025Efficient Inference for Inverse Reinforcement Learning and Dynamic Discrete Choice Models
- 2021Bellman-consistent Pessimism for Offline Reinforcement Learning
- 2025Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning
- 2024Bellman Unbiasedness: Toward Provably Efficient Distributional Reinforcement Learning with General Value Function Approximation
- 2025Implicit Constraint-Aware Off-Policy Correction for Offline Reinforcement Learning
- 2026Inverting the Bellman Equation: From Q-Values to World Models