mlsanityData to 7 Oct 2026

Offline RL algorithms

819 papers1 this weekHigh confidence in this grouping: the papers form one clear topicHigh confidence in this grouping: the papers form one clear topic · About

Keywords: offline, policy, reinforcement learning, reinforcement, offline reinforcement

Key facts. Offline RL algorithms: 819 papers on arXiv since 2018 (data through 7 Oct 2026). 440 of its 819 mapped papers (54%) cite “Conservative Q-Learning for Offline Reinforcement Learning” (2020, arXiv 2006.04779). 416 of its 819 mapped papers (51%) cite “Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems” (2020, arXiv 2005.01643). 10 new papers 17 Sep–7 Oct.

What’s new

17 Sep – 7 Oct 2026

10 new papers, 17 Sep – 7 Oct; 1 this week.

  1. This week · 1
  2. Beyond Policy Support: Interaction Constrained Offline Reinforcement Learning for Autonomous Driving also fits: Imitation and offline RL for driving
  3. Earlier · 9
  4. Role-Adaptive Policy Optimization for Offline Reinforcement Learning also fits: Diffusion and flow policies for offline RL
  5. In-Distribution Imagination for Model-Based Offline Reinforcement Learning also fits: Diffusion and flow policies for offline RL
  6. Finite-Sample Theory for Fitted Q-Iteration When Actions Are Functions also fits: Diffusion and flow policies for offline RL
  7. Deep Weighted Bellman Residual Minimization for Q^* Estimation also fits: Diffusion and flow policies for offline RL
  8. Policy Plasticity Matters in Offline-to-Online Reinforcement Learning: Refitting Offline Policies for Online Adaptation
  9. From Static Policies to Adaptive Priors in Offline Reinforcement Learning
  10. PORL: Pretrained Offline Reinforcement Learning for the Job Shop Scheduling Problem also fits: Offline multi-agent RL
  11. Optimal Sequential Annotations for Off-Policy Evaluation also fits: Safe offline RL
  12. Lifted Bellman Linear Programming for Offline Reinforcement Learning

Foundations

Most cited by papers here

  1. Conservative Q-Learning for Offline Reinforcement Learning2020cited by 440 here
  2. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems2020cited by 416 here
  3. D4RL: Datasets for Deep Data-Driven Reinforcement Learning2020cited by 391 here
  4. A Minimalist Approach to Offline Reinforcement Learning2021cited by 310 here
  5. Offline Reinforcement Learning with Implicit Q-Learning2021cited by 302 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 75 here
  2. 2019An Optimistic Perspective on Offline Reinforcement Learningcited by 131 here
  3. 2019Behavior Regularized Offline Reinforcement Learningcited by 256 here
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 391 here
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 416 here
  6. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 216 here
  7. 2020MOPO: Model-based Offline Policy Optimizationcited by 213 here
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 139 here
  9. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 310 here
  10. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 302 here
  11. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 72 here
13 more it builds on, not in this chain
  1. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 102 here
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 440 here
  3. 2020Critic Regularized Regressioncited by 101 here
  4. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RLcited by 66 here
  5. 2020Is Pessimism Provably Efficient for Offline RL?cited by 140 here
  6. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 171 here
  7. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 123 here
  8. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 89 here
  9. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 65 here
  10. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 103 here
  11. 2021Offline RL Without Off-Policy Evaluationcited by 74 here
  12. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 150 here
  13. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 76 here

The 24 most cited of 804.

Browse

30 of 819, ranked 6–35

  1. 6Behavior Regularized Offline Reinforcement Learning2019256
  2. 7MOReL : Model-Based Offline Reinforcement Learning2020216
  3. 8MOPO: Model-based Offline Policy Optimization2020213
  4. 9COMBO: Conservative Offline Model-Based Policy Optimization2021171
  5. 10Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble2021150
  6. 11Is Pessimism Provably Efficient for Offline RL?2020140
  7. 12An Optimistic Perspective on Offline Reinforcement Learning2019131
  8. 13Offline Reinforcement Learning with Fisher Divergence Critic Regularization2021123
  9. 14Bellman-consistent Pessimism for Offline Reinforcement Learning2021103
  10. 15Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning2020102
All 30
  1. 16Critic Regularized Regression2020101
  2. 17Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism202189
  3. 18Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning202276
  4. 19Offline RL Without Off-Policy Evaluation202174
  5. 20A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems202272
  6. 21EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL202066
  7. 22Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning202165
  8. 23Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble202161
  9. 24Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage202159
  10. 25PLAS: Latent Action Space for Offline Reinforcement Learning202059
  11. 26Adversarially Trained Actor Critic for Offline Reinforcement Learning202258
  12. 27Mildly Conservative Q-Learning for Offline Reinforcement Learning202257
  13. 28What are the Statistical Limits of Offline RL with Linear Function Approximation?202055
  14. 29CORL: Research-oriented Deep Offline Reinforcement Learning Library202254
  15. 30Provably Good Batch Reinforcement Learning Without Great Exploration202054
  16. 31Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning202148
  17. 32Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization202347
  18. 33RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning202247
  19. 34RORL: Robust Offline Reinforcement Learning via Conservative Smoothing202247
  20. 35Batch Value-function Approximation with Only Realizability202046