mlsanityData to 7 Oct 2026

Universal Value Density Estimation for Imitation Learning and Goal-Conditioned Reinforcement Learning

arXiv 2002.06473 ↗First posted 15 Feb 2020cs.LG stat.MLcited by 12

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

This work considers two distinct settings: imitation learning and goal-conditioned reinforcement learning. In either case, effective solutions require the agent to reliably reach a specified state (a goal), or set of states (a demonstration). Drawing a connection between probabilistic long-term dynamics and the desired value function, this work introduces an approach which utilizes recent advances in density estimation to effectively learn to reach a given state. As our first contribution, we use this approach for goal-conditioned reinforcement learning and show that it is both efficient and does not suffer from hindsight bias in stochastic domains. As our second contribution, we extend the approach to imitation learning and show that it achieves state-of-the art demonstration sample-efficiency on standard benchmark tasks.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 23 of its 39 references that are in the map. First posted on arXiv 15 Feb 2020.

Builds on

23 of 39 references.

Closest ancestors · 4

Same direction, newest first

  1. 2019SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards27 May
  2. 2019Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation16 May
  3. 2019Generative predecessor models for sample-efficient imitation learning1 Apr
  4. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep

Other references · 19

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2013Auto-Encoding Variational Bayescited by 3,980
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2016Density estimation using Real NVPcited by 2,029
  5. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  6. 2016OpenAI Gymcited by 1,684
  7. 2016Generative Adversarial Imitation Learningcited by 1,576
  8. 2017Hindsight Experience Replaycited by 1,096
  9. 2016Pixel Recurrent Neural Networkscited by 814
  10. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  11. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  12. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
7 more
  1. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 214
  2. 2018DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skillscited by 153
  3. 2018Zero-Shot Visual Imitationcited by 114
  4. 2018Playing hard exploration games by watching YouTubecited by 98
  5. 2018Temporal Difference Models: Model-Free Deep RL for Model-Based Controlcited by 89
  6. 2019Is the Policy Gradient a Gradient?cited by 40
  7. 2018ARCHER: Aggressive Rewards to Counter bias in Hindsight Experience Replaycited by 9

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 3; 2023: 1; 2024: 3; 2025: 2.

12 papers.

  1. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  2. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 65
  3. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  4. 2024A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoalscited by 20
  5. 2022Hypernetworks for Zero-shot Transfer in Reinforcement Learningcited by 18
  6. 2021Outcome-Driven Reinforcement Learning via Variational Inferencecited by 17
  7. 2021C-Planning: An Automatic Curriculum for Learning Goal-Reaching Taskscited by 15
  8. 2024Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inferencecited by 15
  9. 2025Horizon Generalization in Reinforcement Learningcited by 15
  10. 2023A Coupled Flow Approach to Imitation Learningcited by 12
  11. 2022Goal-Conditioned Q-Learning as Knowledge Distillationcited by 1
  12. 2025Decision Flow Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Probability Density Estimation Based Imitation Learning
  2. 2025Reinforcement Learning via Implicit Imitation Guidance
  3. 2022Imitation Learning by State-Only Distribution Matching
  4. 2020Imitation with Neural Density Models
  5. 2023Conditional Kernel Imitation Learning for Continuous State Environments
  6. 2019Learning to Reach Goals via Iterated Supervised Learning
  7. 2025Value from Observations: Towards Large-Scale Imitation Learning via Self-Improvement
  8. 2019Reward-Conditioned Policies