mlsanityData to 14 Oct 2026

Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning

arXiv 2301.12714 ↗First posted 30 Jan 2023cs.LGcited by 13

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We propose A-Crab (Actor-Critic Regularized by Average Bellman error), a new practical algorithm for offline reinforcement learning (RL) in complex environments with insufficient data coverage. Our algorithm combines the marginalized importance sampling framework with the actor-critic paradigm, where the critic returns evaluations of the actor (policy) that are pessimistic relative to the offline data and have a small average (importance-weighted) Bellman error. Compared to existing methods, our algorithm simultaneously offers a number of advantages: (1) It achieves the optimal statistical rate of 1/√N -- where N is the size of offline dataset -- in converging to the best policy covered in the offline dataset, even when combined with general function approximators. (2) It relies on a weaker average notion of policy coverage (compared to the ℓ_∞ single-policy concentrability) that exploits the structure of policy visitations. (3) It outperforms the data-collection behavior policy over a wide range of specific hyperparameters. We provide both theoretical analysis and experimental results to validate the effectiveness of our proposed algorithm.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 59 of its 81 references that are in the map. First posted on arXiv 30 Jan 2023.

Builds on

59 of 81 references.

Closest ancestors · 34

Same direction, newest first

  1. 2022Offline Reinforcement Learning via Linear-Programming with Error-Bound Induced Constraints28 Dec
  2. 2022ARMOR: A Model-based Framework for Improving Arbitrary Baseline Policies with Offline Data8 Nov
  3. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian1 Nov
  4. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief13 Oct
  5. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity11 Aug
  6. 2022On Gap-dependent Bounds for Offline Reinforcement Learning1 Jun
  7. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  8. 2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gaps25 Mar
  9. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimism11 Mar
  10. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability9 Feb
  11. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  12. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimism17 Oct
22 more
  1. 2021Representation Learning for Online and Offline RL in Low-rank MDPs9 Oct
  2. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning19 Aug
  3. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  4. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation21 Jun
  5. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  6. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  7. 2021Corruption-Robust Offline Reinforcement Learning11 Jun
  8. 2021Offline Reinforcement Learning as Anti-Exploration11 Jun
  9. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  10. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  11. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning10 Mar
  12. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  13. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  14. 2020Batch Value-function Approximation with Only Realizability11 Aug
  15. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  16. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
  17. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  18. 2020MOPO: Model-based Offline Policy Optimization27 May
  19. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  20. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  21. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  22. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 25

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  3. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  4. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  5. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  6. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  7. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  8. 2019Tackling Climate Change with Machine Learningcited by 265
  9. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  10. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  11. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  12. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 159
13 more
  1. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  2. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  3. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 119
  4. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 111
  5. 2020Chip Placement with Deep Reinforcement Learningcited by 83
  6. 2020The Complexity of Constrained Min-Max Optimizationcited by 82
  7. 2020Discovering Reinforcement Learning Algorithmscited by 77
  8. 2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
  9. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  10. 2022The Efficacy of Pessimism in Asynchronous Q-Learningcited by 42
  11. 2014Approximate Policy Iteration Schemes: A Comparisoncited by 39
  12. 2019A Kernel Loss for Solving the Bellman Equationcited by 37
  13. 2019On Value Functions and the Agent-Environment Boundarycited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 3; 2025: 3; 2026: 1.

13 papers.

Sort:

By area:

  1. 2023Offline Primal-Dual Reinforcement Learning for Linear MDPscited by 17
  2. 2023Offline Minimax Soft-Q-learning Under Realizability and Partial Coveragecited by 10
  3. 2025Offline Reinforcement Learning in Large State Spaces: Algorithms and Guaranteescited by 10
  4. 2023A Primal-Dual-Critic Algorithm for Offline Constrained Reinforcement Learningcited by 9
  5. 2023Pessimistic Nonlinear Least-Squares Value Iteration for Offline Reinforcement Learningcited by 9
  6. 2023On Representation Complexity of Model-based and Model-free Reinforcement Learningcited by 6
  7. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyondcited by 6
  8. 2024Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learningcited by 4
  9. 2024The Role of Inherent Bellman Error in Offline Reinforcement Learning with Linear Function Approximationcited by 4
  10. 2023Offline Reinforcement Learning with Additional Covering Distributionscited by 1
  11. 2025Offline Constrained Reinforcement Learning under Partial Data Coveragecited by 1
  12. 2025Avoiding Death through Fear Intrinsic Conditioningcited by 0
1 more
  1. 2026Collaborative Weighting with Pessimistic Critic for Mitigating Overestimation in Off-Policy Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning
  2. 2026Simple Actors and Deep Critics for Scalable Reinforcement Learning
  3. 2025Mildly Conservative Regularized Evaluation for Offline Reinforcement Learning
  4. 2026Flow Actor-Critic for Offline Reinforcement Learning
  5. 2021Offline RL Without Off-Policy Evaluation
  6. 2024Offline Actor-Critic Reinforcement Learning Scales to Large Models
  7. 2023Projected Off-Policy Q-Learning (POP-QL) for Stabilizing Offline Reinforcement Learning
  8. 2021You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL