mlsanityData to 7 Oct 2026

Neural Network Approximation for Pessimistic Offline Reinforcement Learning

arXiv 2312.11863 ↗First posted 19 Dec 2023cs.LG cs.AI stat.MLcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Deep reinforcement learning (RL) has shown remarkable success in specific offline decision-making scenarios, yet its theoretical guarantees are still under development. Existing works on offline RL theory primarily emphasize a few trivial settings, such as linear MDP or general function approximation with strong assumptions and independent data, which lack guidance for practical use. The coupling of deep learning and Bellman residuals makes this problem challenging, in addition to the difficulty of data dependence. In this paper, we establish a non-asymptotic estimation error of pessimistic offline RL using general neural network approximation with C-mixing data regarding the structure of networks, the dimension of datasets, and the concentrability of data coverage, under mild assumptions. Our result shows that the estimation error consists of two parts: the first converges to zero at a desired rate on the sample size with partially controllable concentrability, and the second becomes negligible if the residual constraint is tight. This result demonstrates the explicit efficiency of deep adversarial offline RL frameworks. We utilize the empirical process tool for C-mixing sequences and the neural network approximation theory for the H"older class to achieve this. We also develop methods to bound the Bellman estimation error caused by function approximation with empirical Bellman constraint perturbations. Additionally, we present a result that lessens the curse of dimensionality using data with low intrinsic dimensionality and function classes with low complexity. Our estimation provides valuable insights into the development of deep offline RL and guidance for algorithm model design.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 47 of its 86 references that are in the map. First posted on arXiv 19 Dec 2023.

Builds on

47 of 86 references.

Closest ancestors · 14

Same direction, newest first

  1. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian1 Nov
  2. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability9 Feb
  3. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  4. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  5. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  6. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  7. 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reduction2 Feb
  8. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  9. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?22 Oct
  10. 2020Batch Value-function Approximation with Only Realizability11 Aug
  11. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
2 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  2. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb

Other references · 33

  1. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  2. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
  3. 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
  4. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 307
  5. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 304
  6. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  7. 2019A Theoretical Analysis of Deep Q-Learningcited by 212
  8. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187
  9. 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 165
  10. 2020Deep Network Approximation for Smooth Functionscited by 163
  11. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 160
  12. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 157
21 more
  1. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  2. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  3. 2019Deep Network Approximation Characterized by Number of Neuronscited by 120
  4. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 108
  5. 2019Finite-Sample Analysis for SARSA with Linear Function Approximationcited by 85
  6. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
  7. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  8. 2019Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov spacecited by 46
  9. 2014Approximate Policy Iteration Schemes: A Comparisoncited by 39
  10. 2019Dimensionality compression and expansion in Deep Neural Networkscited by 38
  11. 2019A Kernel Loss for Solving the Bellman Equationcited by 37
  12. 2021Elementary superexpressive activationscited by 35
  13. 2022Approximation bounds for norm constrained neural networks with applications to regression and GANscited by 21
  14. 2019On Value Functions and the Agent-Environment Boundarycited by 19
  15. 2023Deep Operator Learning Lessens the Curse of Dimensionality for PDEscited by 18
  16. 2021On Empirical Risk Minimization with Dependent and Heavy-Tailed Datacited by 16
  17. 2020Learning to Scan: A Deep Reinforcement Learning Approach for Personalized Scanning in CT Imagingcited by 13
  18. 2022Sample Complexity of Nonparametric Off-Policy Evaluation on Low-Dimensional Manifolds using Deep Networkscited by 12
  19. 2016Learning theory estimates with observations from general stationary stochastic processescited by 7
  20. 2020A Kernel Two-sample Test for Dynamical Systemscited by 7
  21. 2022Optimizing Pessimism in Dynamic Treatment Regimes: A Bayesian Learning Approachcited by 7

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025PINS: Proximal Iterations with Sparse Newton and Sinkhorn for Optimal Transportcited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2021Pessimistic Model Selection for Offline Deep Reinforcement Learning
  2. 2021Instabilities of Offline RL with Pre-Trained Neural Representation
  3. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimism
  4. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation
  5. 2026Generalization in offline RL: The structure is more important than the amount of pessimism
  6. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters
  7. 2026Residuals-based Offline Reinforcement Learning
  8. 2021A Minimalist Approach to Offline Reinforcement Learning