mlsanityData to 7 Oct 2026

Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning

arXiv 2108.08812 ↗First posted 19 Aug 2021cs.LGcited by 109

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Actor-critic methods are widely used in offline reinforcement learning practice, but are not so well-understood theoretically. We propose a new offline actor-critic algorithm that naturally incorporates the pessimism principle, leading to several key advantages compared to the state of the art. The algorithm can operate when the Bellman evaluation operator is closed with respect to the action value function of the actor's policies; this is a more general setting than the low-rank MDP model. Despite the added generality, the procedure is computationally tractable as it involves the solution of a sequence of second-order programs. We prove an upper bound on the suboptimality gap of the policy returned by the procedure that depends on the data coverage of any arbitrary, possibly data dependent comparator policy. The achievable guarantee is complemented with a minimax lower bound that is matching up to logarithmic factors.

Key facts. Cited by 109 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 66 of its 100 references that are in the map. First posted on arXiv 19 Aug 2021.

Builds on

66 of 100 references.

Closest ancestors · 16

Same direction, newest first

  1. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  2. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  3. 2021Risk Bounds and Rademacher Complexity in Batch Reinforcement Learning25 Mar
  4. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  5. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  6. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?22 Oct
  7. 2020Batch Value-function Approximation with Only Realizability11 Aug
  8. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
  9. 2020Near-Optimal Provable Uniform Convergence in Offline Policy Evaluation for Reinforcement Learning7 Jul
  10. 2020Critic Regularized Regression26 Jun
  11. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  12. 2020MOPO: Model-based Offline Policy Optimization27 May
4 more
  1. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  3. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  4. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 50

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  3. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  4. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  5. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  6. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  7. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 290
  8. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  9. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  10. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  11. 2019A Theory of Regularized Markov Decision Processescited by 279
  12. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
38 more
  1. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  2. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  3. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 222
  4. 2019A Theoretical Analysis of Deep Q-Learningcited by 212
  5. 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 199
  6. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  7. 2020Learning Near Optimal Policies with Low Inherent Bellman Errorcited by 180
  8. 2021Bilinear Classes: A Structural Framework for Provable Generalization in RLcited by 176
  9. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 174
  10. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 157
  11. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
  12. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  13. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 149
  14. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  15. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  16. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  17. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  18. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 118
  19. 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 113
  20. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 108
  21. 2019Frequentist Regret Bounds for Randomized Least-Squares Value Iterationcited by 100
  22. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  23. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  24. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  25. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
  26. 2020Exponential Lower Bounds for Planning in MDPs With Linearly-Realizable Optimal Action-Value Functionscited by 71
  27. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  28. 2020Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learningcited by 60
  29. 2020Provably Efficient Reward-Agnostic Navigation with Linear Value Iterationcited by 56
  30. 2020Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RLcited by 55
  31. 2013The Information Geometry of Mirror Descentcited by 53
  32. 2021Cautiously Optimistic Policy Optimization and Exploration with Linear Function Approximationcited by 53
  33. 2019Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimationcited by 48
  34. 2021On the Linear convergence of Natural Policy Gradient Algorithmcited by 48
  35. 2019Neural Proximal/Trust Region Policy Optimization Attains Globally Optimal Policycited by 42
  36. 2020Single-Timescale Actor-Critic Provably Finds Globally Optimal Policycited by 34
  37. 2018Representation Balancing MDPs for Off-Policy Policy Evaluationcited by 20
  38. 2021Minimax Model Learningcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 39; 2023: 26; 2024: 22; 2025: 13; 2026: 5.

109 papers.

Sort:

By area:

  1. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  2. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  3. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  4. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  5. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 128
  6. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  7. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  8. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  9. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
  10. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 84
  11. 2022The Role of Coverage in Online Reinforcement Learningcited by 82
  12. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimismcited by 78
38 more
  1. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 71
  2. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  3. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimismcited by 63
  4. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  5. 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Gamecited by 53
  6. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  7. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coveragecited by 51
  8. 2022Pessimistic Minimax Value Iteration: Provably Efficient Equilibrium Learning from Offline Datasetscited by 42
  9. 2022Leveraging Offline Data in Online Reinforcement Learningcited by 42
  10. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 36
  11. 2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gapscited by 34
  12. 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
  13. 2022Policy learning "without" overlap: Pessimism and generalized empirical Bernstein's inequalitycited by 32
  14. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learningcited by 31
  15. 2022Distributionally Robust Offline Reinforcement Learning with Linear Function Approximationcited by 31
  16. 2023Adversarial Model for Offline Reinforcement Learningcited by 31
  17. 2023Corruption-Robust Offline Reinforcement Learning with General Function Approximationcited by 30
  18. 2022Provably Efficient Offline Multi-agent Reinforcement Learning via Strategy-wise Bonuscited by 29
  19. 2022Provable Benefit of Multitask Representation Learning in Reinforcement Learningcited by 29
  20. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangiancited by 29
  21. 2022When is Offline Two-Player Zero-Sum Markov Game Solvable?cited by 27
  22. 2023The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learningcited by 27
  23. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferencescited by 27
  24. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processescited by 26
  25. 2023The Generalization Gap in Offline Reinforcement Learningcited by 25
  26. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
  27. 2022On Instance-Dependent Bounds for Offline Reinforcement Learning with Linear Function Approximationcited by 22
  28. 2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imaginationcited by 21
  29. 2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
  30. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 20
  31. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RLcited by 20
  32. 2022Offline Reinforcement Learning with Differential Privacycited by 19
  33. 2022Pessimism for Offline Linear Contextual Bandits using ℓp Confidence Setscited by 18
  34. 2023Learning in POMDPs is Sample-Efficient with Hindsight Observabilitycited by 17
  35. 2023Offline Primal-Dual Reinforcement Learning for Linear MDPscited by 17
  36. 2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learningcited by 17
  37. 2024Optimal Design for Human Preference Elicitationcited by 17
  38. 2023Offline Learning in Markov Games with General Function Approximationcited by 16

50 most cited of 109 All 109 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Optimization Solution Functions as Deterministic Policies for Offline Reinforcement Learning
  2. 2021You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
  3. 2026Flow Actor-Critic for Offline Reinforcement Learning
  4. 2021Offline RL Without Off-Policy Evaluation
  5. 2026Simple Actors and Deep Critics for Scalable Reinforcement Learning
  6. 2024Offline Actor-Critic Reinforcement Learning Scales to Large Models
  7. 2024Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learning
  8. 2026SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfer