mlsanityData to 7 Oct 2026

Adversarially Trained Actor Critic for Offline Reinforcement Learning

arXiv 2202.02446 ↗First posted 5 Feb 2022cs.LGcited by 111

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We propose Adversarially Trained Actor Critic (ATAC), a new model-free algorithm for offline reinforcement learning (RL) under insufficient data coverage, based on the concept of relative pessimism. ATAC is designed as a two-player Stackelberg game: A policy actor competes against an adversarially trained value critic, who finds data-consistent scenarios where the actor is inferior to the data-collection behavior policy. We prove that, when the actor attains no regret in the two-player game, running ATAC produces a policy that provably 1) outperforms the behavior policy over a wide range of hyperparameters that control the degree of pessimism, and 2) competes with the best policy covered by data with appropriately chosen hyperparameters. Compared with existing works, notably our framework offers both theoretical guarantees for general function approximation and a deep RL implementation scalable to complex environments and large datasets. In the D4RL benchmark, ATAC consistently outperforms state-of-the-art offline RL algorithms on a range of continuous control tasks.

Key facts. Cited by 111 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 31 of its 58 references that are in the map. First posted on arXiv 5 Feb 2022.

Builds on

31 of 58 references.

Closest ancestors · 16

Same direction, newest first

  1. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learning26 Oct
  2. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  3. 2021Representation Learning for Online and Offline RL in Low-rank MDPs9 Oct
  4. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning19 Aug
  5. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  6. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  7. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  8. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  9. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  10. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
  11. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  12. 2020MOPO: Model-based Offline Policy Optimization27 May
4 more
  1. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  3. 2020Q* Approximation Schemes for Batch Reinforcement Learning: A Theoretical Comparison9 Mar
  4. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 15

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  7. 2019A Theory of Regularized Markov Decision Processescited by 279
  8. 2017A unified view of entropy-regularized Markov decision processescited by 174
  9. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  10. 2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
  11. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 80
  12. 2020A Game Theoretic Framework for Model Based Reinforcement Learningcited by 72
3 more
  1. 2021Heuristic-Guided Reinforcement Learningcited by 41
  2. 2021Stackelberg Actor-Critic: Game-Theoretic Reinforcement Learning Algorithmscited by 18
  3. 2021Convergent and Efficient Deep Q Network Algorithmcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 44; 2024: 32; 2025: 14; 2026: 8.

111 papers.

Sort:

By area:

  1. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  2. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  3. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  4. 2023On Penalty-based Bilevel Gradient Descent Methodcited by 87
  5. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 83
  6. 2022The Role of Coverage in Online Reinforcement Learningcited by 82
  7. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
  8. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  9. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  10. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coveragecited by 51
  11. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  12. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
38 more
  1. 2023Behavior Proximal Policy Optimizationcited by 43
  2. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 36
  3. 2022Distributionally Robust Offline Reinforcement Learning with Linear Function Approximationcited by 31
  4. 2022Data-Driven Offline Decision-Making via Invariant Representation Learningcited by 31
  5. 2023Adversarial Model for Offline Reinforcement Learningcited by 31
  6. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangiancited by 29
  7. 2023The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learningcited by 27
  8. 2023When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learningcited by 26
  9. 2022Hindsight Learning for MDPs with Exogenous Inputscited by 25
  10. 2023The Generalization Gap in Offline Reinforcement Learningcited by 25
  11. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
  12. 2023Benchmarking Offline Reinforcement Learning on Real-Robot Hardwarecited by 23
  13. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 23
  14. 2022On Instance-Dependent Bounds for Offline Reinforcement Learning with Linear Function Approximationcited by 22
  15. 2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learningcited by 21
  16. 2024Doubly Mild Generalization for Offline Reinforcement Learningcited by 21
  17. 2023Survival Instinct in Offline Reinforcement Learningcited by 20
  18. 2023Supported Trust Region Optimization for Offline Reinforcement Learningcited by 20
  19. 2022Pessimism for Offline Linear Contextual Bandits using ℓp Confidence Setscited by 18
  20. 2023MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observationscited by 18
  21. 2022When is Realizability Sufficient for Off-Policy Reinforcement Learning?cited by 17
  22. 2023Offline Primal-Dual Reinforcement Learning for Linear MDPscited by 17
  23. 2024Offline Reinforcement Learning with OOD State Correction and OOD Action Suppressioncited by 17
  24. 2022Dungeons and Data: A Large-Scale NetHack Datasetcited by 16
  25. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Modelscited by 16
  26. 2023Mind the Gap: Offline Policy Optimization for Imperfect Rewardscited by 15
  27. 2024SEABO: A Simple Search-Based Method for Offline Imitation Learningcited by 14
  28. 2024Latent Plan Transformer for Trajectory Abstraction: Planning as Latent Space Inferencecited by 14
  29. 2023Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learningcited by 13
  30. 2023Uncertainty-driven Trajectory Truncation for Data Augmentation in Offline Reinforcement Learningcited by 13
  31. 2023The Virtues of Laziness in Model-based RL: A Unified Objective and Algorithmscited by 12
  32. 2023A Survey of Demonstration Learningcited by 12
  33. 2024D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learningcited by 11
  34. 2023Safe Policy Improvement for POMDPs via Finite-State Controllerscited by 10
  35. 2023Improving Offline RL by Blending Heuristicscited by 10
  36. 2024Hokoff: Real Game Dataset from Honor of Kings and its Offline Reinforcement Learning Benchmarkscited by 10
  37. 2025Offline Reinforcement Learning in Large State Spaces: Algorithms and Guaranteescited by 10
  38. 2022ARMOR: A Model-based Framework for Improving Arbitrary Baseline Policies with Offline Datacited by 9

50 most cited of 111 All 111 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Simple Actors and Deep Critics for Scalable Reinforcement Learning
  2. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL
  3. 2024Offline Actor-Critic Reinforcement Learning Scales to Large Models
  4. 2026Decoupling Policy Extraction for Offline Reinforcement Learning
  5. 2026Flow Actor-Critic for Offline Reinforcement Learning
  6. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning
  7. 2026SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfer
  8. 2021You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL