mlsanityData to 7 Oct 2026

Adversarial Model for Offline Reinforcement Learning

arXiv 2302.11048 ↗First posted 21 Feb 2023cs.LG cs.AIcited by 31

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We propose a novel model-based offline Reinforcement Learning (RL) framework, called Adversarial Model for Offline Reinforcement Learning (ARMOR), which can robustly learn policies to improve upon an arbitrary reference policy regardless of data coverage. ARMOR is designed to optimize policies for the worst-case performance relative to the reference policy through adversarially training a Markov decision process model. In theory, we prove that ARMOR, with a well-tuned hyperparameter, can compete with the best policy within data coverage when the reference policy is supported by the data. At the same time, ARMOR is robust to hyperparameter choices: the policy learned by ARMOR, with "any" admissible hyperparameter, would never degrade the performance of the reference policy, even when the reference policy is not covered by the dataset. To validate these properties in practice, we design a scalable implementation of ARMOR, which by adversarial training, can optimize policies without using model ensembles in contrast to typical model-based methods. We show that ARMOR achieves competent performance with both state-of-the-art offline model-free and model-based RL algorithms and can robustly improve the reference policy over various hyperparameter choices.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 34 of its 48 references that are in the map. First posted on arXiv 21 Feb 2023.

Builds on

34 of 48 references.

Closest ancestors · 20

Same direction, newest first

  1. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  2. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity28 Feb
  3. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  4. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  5. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning19 Aug
  6. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  7. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  8. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  9. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning9 Jun
  10. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  11. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  12. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
8 more
  1. 2020Batch Value-function Approximation with Only Realizability11 Aug
  2. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  3. 2020MOPO: Model-based Offline Policy Optimization27 May
  4. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  7. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  8. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 14

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2023Mastering Diverse Domains through World Modelscited by 1,193
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  7. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  8. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  9. 2018Residual Policy Learningcited by 146
  10. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  11. 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
  12. 2019Off-Policy Policy Gradient with State Distribution Correctioncited by 43
2 more
  1. 2021On the Optimality of Batch Policy Optimization Algorithmscited by 28
  2. 2022Adversarial Counterfactual Environment Model Learningcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 8; 2025: 12; 2026: 4.

31 papers.

Sort:

By area:

  1. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  2. 2023Survival Instinct in Offline Reinforcement Learningcited by 20
  3. 2024Refining Minimax Regret for Unsupervised Environment Designcited by 16
  4. 2025Offline Reinforcement Learning in Large State Spaces: Algorithms and Guaranteescited by 10
  5. 2023MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operatorcited by 9
  6. 2023A Survey on Offline Model-Based Reinforcement Learningcited by 7
  7. 2025Outcome-Based Online Reinforcement Learning: Algorithms and Fundamental Limitscited by 7
  8. 2024Provably and Practically Efficient Adversarial Imitation Learning with General Function Approximationcited by 6
  9. 2025Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learningcited by 6
  10. 2025Learning a Pessimistic Reward Model in RLHFcited by 5
  11. 2023DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learningcited by 4
  12. 2024Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learningcited by 4
19 more
  1. 2024Universal Black-Box Reward Poisoning Attack against Offline Reinforcement Learningcited by 4
  2. 2023Achieving the Asymptotically Optimal Sample Complexity of Offline Reinforcement Learning: A DRO-Based Approachcited by 3
  3. 2023CROP: Conservative Reward for Model-based Offline Policy Optimizationcited by 3
  4. 2025Adversarial Policy Optimization for Offline Preference-based Reinforcement Learningcited by 2
  5. 2025Towards Optimal Offline Reinforcement Learningcited by 2
  6. 2025Model-Based Offline Reinforcement Learning with Adversarial Data Augmentationcited by 2
  7. 2026Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weightingcited by 2
  8. 2026Theoretical Foundations and Effective Algorithms for Policy-Aware Simulator Learningcited by 2
  9. 2024Real-World Fluid Directed Rigid Body Control via Deep Reinforcement Learningcited by 1
  10. 2024Offline Model-Based Reinforcement Learning with Anti-Explorationcited by 1
  11. 2025Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspectivecited by 1
  12. 2025Offline Reinforcement Learning using Human-Aligned Reward Labeling for Autonomous Emergency Braking in Occluded Pedestrian Crossingcited by 1
  13. 2025Imagination-Limited Q-Learning for Offline Reinforcement Learningcited by 1
  14. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
  15. 2026Adversarial Imitation Learning with General Function Approximation: Theoretical Analysis and Practical Algorithmscited by 1
  16. 2024MoMA: Model-based Mirror Ascent for Offline Reinforcement Learningcited by 0
  17. 2024Stochastic Parrots or ICU Experts? Large Language Models in Critical Care Medicine: A Scoping Reviewcited by 0
  18. 2025Large Language Model-Enhanced Reinforcement Learning for Diverse and Novel Recommendationscited by 0
  19. 2026Neuro-Inspired Inverse Learning for Planning and Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022ARMOR: A Model-based Framework for Improving Arbitrary Baseline Policies with Offline Data
  2. 2024Towards Robust Policy: Enhancing Offline Reinforcement Learning with Adversarial Attacks and Defenses
  3. 2023Model-based Offline Policy Optimization with Adversarial Network
  4. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  5. 2022Domain Generalization for Robust Model-Based Offline Reinforcement Learning
  6. 2024Improving Offline Reinforcement Learning with Inaccurate Simulators
  7. 2025A Clean Slate for Offline Reinforcement Learning
  8. 2021Online and Offline Reinforcement Learning by Planning with a Learned Model