mlsanityData to 14 Oct 2026

Offline Reinforcement Learning with Value-based Episodic Memory

arXiv 2110.09796 ↗First posted 19 Oct 2021cs.LG cs.AIcited by 36

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) shows promise of applying RL to real-world problems by effectively utilizing previously collected data. Most existing offline RL algorithms use regularization or constraints to suppress extrapolation error for actions outside the dataset. In this paper, we adopt a different framework, which learns the V-function instead of the Q-function to naturally keep the learning procedure within the support of an offline dataset. To enable effective generalization while maintaining proper conservatism in offline learning, we propose Expectile V-Learning (EVL), which smoothly interpolates between the optimal value learning and behavior cloning. Further, we introduce implicit planning along offline trajectories to enhance learned V-values and accelerate convergence. Together, we present a new offline method called Value-based Episodic Memory (VEM). We provide theoretical analysis for the convergence properties of our proposed VEM method, and empirical results in the D4RL benchmark show that our method achieves superior performance in most tasks, particularly in sparse-reward tasks.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 33 of its 35 references that are in the map. First posted on arXiv 19 Oct 2021.

Builds on

33 of 35 references.

Closest ancestors · 14

Same direction, newest first

  1. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation21 Jun
  2. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  3. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  4. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  5. 2021Instabilities of Offline RL with Pre-Trained Neural Representation8 Mar
  6. 2021Offline Reinforcement Learning with Pseudometric Learning2 Mar
  7. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  8. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  9. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  10. 2020MOPO: Model-based Offline Policy Optimization27 May
  11. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
2 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  2. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb

Other references · 19

  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  3. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  4. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  5. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  6. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  7. 2019A Theory of Regularized Markov Decision Processescited by 286
  8. 2017Neural Episodic Controlcited by 151
  9. 2020Model-Based Offline Planningcited by 129
  10. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  11. 2016Model-Free Episodic Controlcited by 102
  12. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 91
7 more
  1. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 79
  2. 2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
  3. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 63
  4. 2018Episodic Memory Deep Q-Networkscited by 42
  5. 2021Generalizable Episodic Memory for Deep Reinforcement Learningcited by 23
  6. 2019Adaptive Trade-Offs in Off-Policy Learningcited by 20
  7. 2020Self-Imitation Learning via Generalized Lower Bound Q-learningcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 10; 2023: 15; 2024: 6; 2025: 3; 2026: 2.

36 papers.

Sort:

By area:

  1. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  2. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  3. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  4. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 84
  5. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 64
  6. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 59
  7. 2023Boosting Continuous Control with Consistency Policycited by 38
  8. 2023The In-Sample Softmax for Offline Reinforcement Learningcited by 33
  9. 2023DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learningcited by 28
  10. 2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imaginationcited by 21
  11. 2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
  12. 2024Doubly Mild Generalization for Offline Reinforcement Learningcited by 21
24 more
  1. 2024AlignIQL: Policy Alignment in Implicit Q-Learning through Constrained Optimizationcited by 19
  2. 2022On the Role of Discount Factor in Offline Reinforcement Learningcited by 16
  3. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learningcited by 14
  4. 2023Unsupervised Behavior Extraction via Random Intent Priorscited by 13
  5. 2022Flow to Control: Offline Reinforcement Learning with Lossless Primitive Discoverycited by 12
  6. 2023The Provable Benefits of Unsupervised Data Sharing for Offline Reinforcement Learningcited by 10
  7. 2023Understanding and Addressing the Pitfalls of Bisimulation-based Representations in Offline Reinforcement Learningcited by 10
  8. 2023Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examplescited by 9
  9. 2023Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimizationcited by 8
  10. 2023Neural Episodic Control with State Abstractioncited by 7
  11. 2023Learning Diverse Risk Preferences in Population-based Self-playcited by 6
  12. 2023No Prior Mask: Eliminate Redundant Action for Deep Reinforcement Learningcited by 6
  13. 2024Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replaycited by 6
  14. 2022State Advantage Weighting for Offline RLcited by 5
  15. 2025Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Datasetcited by 4
  16. 2023Swapped goal-conditioned offline reinforcement learningcited by 3
  17. 2025Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruptioncited by 3
  18. 2026Uncertainty-Aware Rank-One MIMO Q Network Framework for Accelerated Offline Reinforcement Learningcited by 3
  19. 2025Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learningcited by 2
  20. 2023HIPODE: Enhancing Offline Reinforcement Learning with High-Quality Synthetic Data from a Policy-Decoupled Approachcited by 1
  21. 2023Model-enhanced Contrastive Reinforcement Learning for Sequential Recommendationcited by 1
  22. 2022DCE: Offline Reinforcement Learning With Double Conservative Estimatescited by 0
  23. 2024Hierarchical Multi-agent Meta-Reinforcement Learning for Cross-channel Biddingcited by 0
  24. 2026OPRIDE: Offline Preference-based Reinforcement Learning via In-Dataset Explorationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021A Minimalist Approach to Offline Reinforcement Learning
  2. 2024Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning
  3. 2026Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning
  4. 2024Is Value Learning Really the Main Bottleneck in Offline RL?
  5. 2025A Clean Slate for Offline Reinforcement Learning
  6. 2026STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order
  7. 2024Offline-to-Online Multi-Agent Reinforcement Learning with Offline Value Function Memory and Sequential Exploration
  8. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory