# Curiosity-Driven Experience Prioritization via Density Estimation

arXiv: https://arxiv.org/abs/1902.08039 · HTML: https://mlsanity.com/paper/1902.08039/

## Key facts

- Cited by 27 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 20 Feb 2019.

## Built on by (most cited first, 27 of 27)

- [Exploration in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2205.00824/) (2022)
- [An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey](https://mlsanity.com/paper/2209.08890/) (2022)
- [Sample-efficient Reinforcement Learning Representation Learning with Curiosity Contrastive Forward Dynamics Model](https://mlsanity.com/paper/2103.08255/) (2021)
- [From Psychological Curiosity to Artificial Curiosity: Curiosity-Driven Learning in Artificial Intelligence Tasks](https://mlsanity.com/paper/2201.08300/) (2022)
- [CCLF: A Contrastive-Curiosity-Driven Learning Framework for Sample-Efficient Reinforcement Learning](https://mlsanity.com/paper/2205.00943/) (2022)
- [Curiosity in Hindsight: Intrinsic Exploration in Stochastic Environments](https://mlsanity.com/paper/2211.10515/) (2022)
- [Improving Environment Novelty Quantification for Effective Unsupervised Environment Design](https://mlsanity.com/paper/2502.05726/) (2025)
- [Null Counterfactual Factor Interactions for Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2505.03172/) (2025)
- [Hindsight Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1903.07854/) (2019)
- [Boosting Exploration in Actor-Critic Algorithms by Incentivizing Plausible Novel States](https://mlsanity.com/paper/2210.00211/) (2022)
- [Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Review](https://mlsanity.com/paper/2411.10268/) (2024)
- [Look Before Leap: Look-Ahead Planning with Uncertainty in Reinforcement Learning](https://mlsanity.com/paper/2503.20139/) (2025)
- [Fixed β-VAE Encoding for Curious Exploration in Complex 3D Environments](https://mlsanity.com/paper/2105.08568/) (2021)
- [Improved Exploring Starts by Kernel Density Estimation-Based State-Space Coverage Acceleration in Reinforcement Learning](https://mlsanity.com/paper/2105.08990/) (2021)
- [Efficient Sparse-Reward Goal-Conditioned Reinforcement Learning with a High Replay Ratio and Regularization](https://mlsanity.com/paper/2312.05787/) (2023)
- [Boosting Efficiency in Task-Agnostic Exploration through Causal Knowledge](https://mlsanity.com/paper/2407.20506/) (2024)
- [SENIOR: Efficient Query Selection and Preference-Guided Exploration in Preference-based Reinforcement Learning](https://mlsanity.com/paper/2506.14648/) (2025)
- [Mixture of Autoencoder Experts Guidance using Unlabeled and Incomplete Data for Exploration in Reinforcement Learning](https://mlsanity.com/paper/2507.15287/) (2025)
- [Knowledge is reward: Learning optimal exploration by predictive reward cashing](https://mlsanity.com/paper/2109.08518/) (2021)
- [Exploring More When It Needs in Deep Reinforcement Learning](https://mlsanity.com/paper/2109.13477/) (2021)
- [Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction](https://mlsanity.com/paper/2110.12081/) (2021)
- [GAN-based Intrinsic Exploration For Sample Efficient Reinforcement Learning](https://mlsanity.com/paper/2206.14256/) (2022)
- [Quantification before Selection: Active Dynamics Preference for Robust Reinforcement Learning](https://mlsanity.com/paper/2209.11596/) (2022)
- [VDSC: Enhancing Exploration Timing with Value Discrepancy and State Counts](https://mlsanity.com/paper/2403.17542/) (2024)
- [Adventurer: Exploration with BiGAN for Deep Reinforcement Learning](https://mlsanity.com/paper/2503.18612/) (2025)
- [VLM-Guided Experience Replay](https://mlsanity.com/paper/2602.01915/) (2026)
- [Integrating Novelty and Surprise for Experience Prioritization and Exploration in Image-Based Reinforcement Learning](https://mlsanity.com/paper/2608.17373/) (2026)

## Abstract (arXiv)

In Reinforcement Learning (RL), an agent explores the environment and collects trajectories into the memory buffer for later learning. However, the collected trajectories can easily be imbalanced with respect to the achieved goal states. The problem of learning from imbalanced data is a well-known problem in supervised learning, but has not yet been thoroughly researched in RL. To address this problem, we propose a novel Curiosity-Driven Prioritization (CDP) framework to encourage the agent to over-sample those trajectories that have rare achieved goal states. The CDP framework mimics the human learning process and focuses more on relatively uncommon events. We evaluate our methods using the robotic environment provided by OpenAI Gym. The environment contains six robot manipulation tasks. In our experiments, we combined CDP with Deep Deterministic Policy Gradient (DDPG) with or without Hindsight Experience Replay (HER). The experimental results show that CDP improves both performance and sample-efficiency of reinforcement learning agents, compared to state-of-the-art methods.