mlsanityData to 14 Oct 2026

Expert Behavior Prior Reinforcement Learning

arXiv 2607.21302 ↗First posted 23 Jul 2026cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Behavior prior reinforcement learning (BPRL) has emerged as a promising paradigm to improve sample efficiency in online reinforcement learning (RL) by leveraging policy priors derived from offline demonstrations. However, most existing BPRL methods rely on static offline datasets, which often suffer from low data diversity and suboptimal trajectory quality. This reliance restricts the effectiveness of policy priors, hindering both policy exploitation and stability during online training. Consequently, agents are prone to inefficient exploration and unstable learning dynamics. To address these limitations, we deviate from existing offline pre-training methods and propose an Expert Behavior Prior (EBP) algorithm. Specifically, we introduce a Q-guided conditional variational autoencoder (Q-CVAE) that learns to generate expert policy priors directly from the online replay buffer. This enables the generation of high-value actions for guiding policy updates without relying on pre-collected expert trajectories. To further enhance policy exploitation, we propose an expert policy guidance (EPG) mechanism that selects expert actions from a generative support set, and we integrate a policy gradient correction (PGC) module to harmonize Q-guidance with expert supervision, promoting stable and consistent policy improvement. Extensive experiments conducted on robotic control (Gym, PyBullet) and industrial control (DMControl) benchmarks demonstrate that EBP significantly outperforms state-of-the-art online RL algorithms, achieving higher sample efficiency and more stable convergence.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 47 of its 64 references that are in the map. First posted on arXiv 23 Jul 2026.

Builds on

47 of 64 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Improving Policy Exploitation in Online Reinforcement Learning with Instant Retrospect Action27 Jan
  2. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL28 May

Other references · 45

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  8. 2018Is Q-learning Provably Efficient?cited by 487
  9. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  10. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  11. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 148
  12. 2022Jump-Start Reinforcement Learningcited by 112
33 more
  1. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  2. 2024Logit Standardization in Knowledge Distillationcited by 102
  3. 2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 85
  4. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 76
  5. 2022Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstrationcited by 54
  6. 2022Leveraging Offline Data in Online Reinforcement Learningcited by 42
  7. 2020Behavior Priors for Efficient Reinforcement Learningcited by 41
  8. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learningcited by 40
  9. 2018Q-learning with Nearest Neighborscited by 37
  10. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learningcited by 34
  11. 2020Softmax Deep Double Deterministic Policy Gradientscited by 32
  12. 2023Policy Gradient for Rectangular Robust Markov Decision Processescited by 31
  13. 2022Don't Start From Scratch: Leveraging Prior Data to Automate Robotic Reinforcement Learningcited by 29
  14. 2022Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Roboticscited by 24
  15. 2023Accelerating Exploration with Unlabeled Prior Datacited by 21
  16. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
  17. 2023Reparameterized Policy Learning for Multimodal Trajectory Optimizationcited by 20
  18. 2023One ACT Play: Single Demonstration Behavior Cloning with Action Chunking Transformerscited by 17
  19. 2023Leveraging Demonstrations to Improve Online Learning: Quality Matterscited by 13
  20. 2024Leveraging Skills from Unlabeled Prior Data for Efficient Online Explorationcited by 13
  21. 2024Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learningcited by 13
  22. 2022Behavior Prior Representation learning for Offline Reinforcement Learningcited by 12
  23. 2021Theoretically Principled Deep RL Acceleration via Nearest Neighbor Function Approximationcited by 11
  24. 2025X-IL: Exploring the Design Space of Imitation Learning Policiescited by 10
  25. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RLcited by 9
  26. 2022Frustratingly Easy Regularization on Representation Can Boost Deep Reinforcement Learningcited by 8
  27. 2025Reinforcement Learning with Stochastic Reward Machinescited by 7
  28. 2022Diverse Imitation Learning via Self-Organizing Generative Modelscited by 5
  29. 2023Learning Sparse Control Tasks from Pixels by Latent Nearest-Neighbor-Guided Explorationscited by 4
  30. 2024Enhancing Reinforcement Learning Agents with Local Guidescited by 4
  31. 2024DIDI: Diffusion-Guided Diversity for Offline Behavioral Generationcited by 4
  32. 2024BLEND: Behavior-guided Neural Population Dynamics Modeling via Privileged Knowledge Distillationcited by 4
  33. 2024Markov Balance Satisfaction Improves Performance in Strictly Batch Offline Imitation Learningcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Improving Online Reinforcement Learning via Bidirectional Behavior Prior Distillation
  2. 2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation
  3. 2025Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RL
  4. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
  5. 2024Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning
  6. 2025A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learning
  7. 2025Prior-Guided Diffusion Planning for Offline Reinforcement Learning
  8. 2024A Model-Based Approach for Improving Reinforcement Learning Efficiency Leveraging Expert Observations