mlsanityData to 7 Oct 2026

ARMOR: A Model-based Framework for Improving Arbitrary Baseline Policies with Offline Data

arXiv 2211.04538 ↗First posted 8 Nov 2022cs.LG cs.AIcited by 9

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We propose a new model-based offline RL framework, called Adversarial Models for Offline Reinforcement Learning (ARMOR), which can robustly learn policies to improve upon an arbitrary baseline policy regardless of data coverage. Based on the concept of relative pessimism, ARMOR is designed to optimize for the worst-case relative performance when facing uncertainty. In theory, we prove that the learned policy of ARMOR never degrades the performance of the baseline policy with any admissible hyperparameter, and can learn to compete with the best policy within data coverage when the hyperparameter is well tuned, and the baseline policy is supported by the data. Such a robust policy improvement property makes ARMOR especially suitable for building real-world learning systems, because in practice ensuring no performance degradation is imperative before considering any benefit learning can bring.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 17 of its 21 references that are in the map. First posted on arXiv 8 Nov 2022.

Builds on

17 of 21 references.

Closest ancestors · 11

Same direction, newest first

  1. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  2. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  3. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  4. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  5. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning9 Jun
  6. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  7. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  9. 2020MOPO: Model-based Offline Policy Optimization27 May
  10. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  11. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 6

  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  3. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  4. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  5. 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
  6. 2021On the Optimality of Batch Policy Optimization Algorithmscited by 28

Built on by

Cited by year

Citing papers per year: 2023: 6; 2024: 1; 2025: 2.

9 papers.

  1. 2023Survival Instinct in Offline Reinforcement Learningcited by 20
  2. 2023MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observationscited by 18
  3. 2025Do We Need to Verify Step by Step? Rethinking Process Supervision from a Theoretical Perspectivecited by 16
  4. 2023Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learningcited by 13
  5. 2023Near-optimal Conservative Exploration in Reinforcement Learning under Episode-wise Constraintscited by 4
  6. 2023Budgeting Counterfactual for Offline RLcited by 4
  7. 2025Balancing optimism and pessimism in offline-to-online learningcited by 4
  8. 2024Trajectory Data Suffices for Statistically Efficient Learning in Offline RL with Linear qπ-Realizability and Concentrabilitycited by 3
  9. 2023Bayesian Regret Minimization in Offline Banditscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Adversarial Model for Offline Reinforcement Learning
  2. 2024Towards Robust Policy: Enhancing Offline Reinforcement Learning with Adversarial Attacks and Defenses
  3. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning
  4. 2023Model-based Offline Policy Optimization with Adversarial Network
  5. 2025Model-Based Offline Reinforcement Learning with Adversarial Data Augmentation
  6. 2022Domain Generalization for Robust Model-Based Offline Reinforcement Learning
  7. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias
  8. 2020POPO: Pessimistic Offline Policy Optimization