mlsanityData to 7 Oct 2026

CEIP: Combining Explicit and Implicit Priors for Reinforcement Learning with Demonstrations

arXiv 2210.09496 ↗First posted 18 Oct 2022cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Although reinforcement learning has found widespread use in dense reward settings, training autonomous agents with sparse rewards remains challenging. To address this difficulty, prior work has shown promising results when using not only task-specific demonstrations but also task-agnostic albeit somewhat related demonstrations. In most cases, the available demonstrations are distilled into an implicit prior, commonly represented via a single deep net. Explicit priors in the form of a database that can be queried have also been shown to lead to encouraging results. To better benefit from available demonstrations, we develop a method to Combine Explicit and Implicit Priors (CEIP). CEIP exploits multiple implicit priors in the form of normalizing flows in parallel to form a single complex prior. Moreover, CEIP uses an effective explicit retrieval and push-forward mechanism to condition the implicit priors. In three challenging environments, we find the proposed CEIP method to improve upon sophisticated state-of-the-art techniques.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 46 of its 59 references that are in the map. First posted on arXiv 18 Oct 2022.

Builds on

46 of 59 references.

References · 46

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2016Density estimation using Real NVPcited by 2,029
  6. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  8. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  9. 2017Masked Autoregressive Flow for Density Estimationcited by 680
  10. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  11. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 636
  12. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
34 more
  1. 2019Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examplescited by 312
  2. 2019Learning Latent Plans from Playcited by 310
  3. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  4. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  5. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  6. 2018Neural Autoregressive Flowscited by 207
  7. 2020Neural Topological SLAM for Visual Navigationcited by 193
  8. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
  9. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 149
  10. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
  11. 2019Meta-Learning with Warped Gradient Descentcited by 98
  12. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  13. 2021RotoGrad: Gradient Homogenization in Multitask Learningcited by 82
  14. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulationcited by 75
  15. 2021Demonstration-Guided Reinforcement Learning with Learned Skillscited by 73
  16. 2018Sylvester Normalizing Flows for Variational Inferencecited by 71
  17. 2020A Universal Representation Transformer Layer for Few-Shot Image Classificationcited by 65
  18. 2018Generative Adversarial User Model for Reinforcement Learning Based Recommendation Systemcited by 64
  19. 2021Learning a Universal Template for Few-shot Dataset Generalizationcited by 61
  20. 2022Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstrationcited by 54
  21. 2020Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learningcited by 53
  22. 2020Scalable Transfer Learning with Expert Modelscited by 48
  23. 2020ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulationcited by 47
  24. 2019Leveraging exploration in off-policy algorithms via normalizing flowscited by 45
  25. 2021TRAIL: Near-Optimal Imitation Learning with Suboptimal Datacited by 39
  26. 2019Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policiescited by 37
  27. 2021Hierarchical Few-Shot Imitation with Skill Transition Modelscited by 37
  28. 2022Skill-based Meta-Reinforcement Learningcited by 36
  29. 2018Randomized Value Functions via Multiplicative Normalizing Flowscited by 15
  30. 2020Variational Mixture of Normalizing Flowscited by 15
  31. 2022IL-flOw: Imitation Learning from Observation using Normalizing Flowscited by 12
  32. 2018Boosting Trust Region Policy Optimization by Normalizing Flows Policycited by 11
  33. 2021Action Priors for Large Action Spaces in Roboticscited by 10
  34. 2020Learning Stable Normalizing-Flow Control for Robotic Manipulationcited by 9

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 1.

4 papers.

  1. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  2. 2024Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformerscited by 8
  3. 2023No Prior Mask: Eliminate Redundant Action for Deep Reinforcement Learningcited by 6
  4. 2025Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Datacited by 0