mlsanityData to 7 Oct 2026

Learning from Sparse Offline Datasets via Conservative Density Estimation

arXiv 2401.08819 ↗First posted 16 Jan 2024cs.LG cs.AIcited by 9

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) offers a promising direction for learning policies from pre-collected datasets without requiring further interactions with the environment. However, existing methods struggle to handle out-of-distribution (OOD) extrapolation errors, especially in sparse reward or scarce data settings. In this paper, we propose a novel training algorithm called Conservative Density Estimation (CDE), which addresses this challenge by explicitly imposing constraints on the state-action occupancy stationary distribution. CDE overcomes the limitations of existing approaches, such as the stationary distribution correction method, by addressing the support mismatch issue in marginal importance sampling. Our method achieves state-of-the-art performance on the D4RL benchmark. Notably, CDE consistently outperforms baselines in challenging tasks with sparse rewards or insufficient data, demonstrating the advantages of our approach in addressing the extrapolation error problem in offline RL.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 38 of its 50 references that are in the map. First posted on arXiv 16 Jan 2024.

Builds on

38 of 50 references.

Closest ancestors · 19

Same direction, newest first

  1. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting22 Jun
  2. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  3. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learning11 Apr
  4. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  5. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity28 Feb
  6. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability9 Feb
  7. 2021d3rlpy: An Offline Deep Reinforcement Learning Library6 Nov
  8. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  9. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation21 Jun
  10. 2021Offline RL Without Off-Policy Evaluation16 Jun
  11. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  12. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
7 more
  1. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning9 Jun
  2. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  3. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  7. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 19

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  3. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  4. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  5. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  7. 2019Solving Rubik's Cube with a Robot Handcited by 699
  8. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  9. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  10. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  11. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  12. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
7 more
  1. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  2. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  3. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  4. 2020Error Bounds of Imitating Policies and Environmentscited by 90
  5. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  6. 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
  7. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 65

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 2; 2026: 2.

9 papers.

  1. 2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learningcited by 10
  2. 2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learningcited by 7
  3. 2024Generalize by Touching: Tactile Ensemble Skill Transfer for Robotic Furniture Assemblycited by 6
  4. 2026ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulationcited by 4
  5. 2024Feasibility Consistent Representation Learning for Safe Reinforcement Learningcited by 3
  6. 2025Video-Enhanced Offline Reinforcement Learning: A Model-Based Approachcited by 1
  7. 2024Imitating from auxiliary imperfect demonstrations via Adversarial Density Weighted Regressioncited by 0
  8. 2025A Comparison Between Decision Transformers and Traditional Offline Reinforcement Learning Algorithmscited by 0
  9. 2026Lifted Bellman Linear Programming for Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024CDSA: Conservative Denoising Score-based Algorithm for Offline Reinforcement Learning
  2. 2022DCE: Offline Reinforcement Learning With Double Conservative Estimates
  3. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
  4. 2023Conservative State Value Estimation for Offline Reinforcement Learning
  5. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian
  6. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing
  7. 2021Reducing Conservativeness Oriented Offline Reinforcement Learning
  8. 2024Strategically Conservative Q-Learning