mlsanityData to 14 Oct 2026

Adaptive Expert Guidance for Efficient On-Policy Reinforcement Learning

arXiv 2610.06019 ↗First posted 5 Oct 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations also fits: RL for games and combinatorial optimization

Abstract (arXiv)

With massively parallel simulation, on-policy Reinforcement Learning methods such as PPO have become standard in many domains. However, learning from scratch is sample-inefficient and fails to exploit the potential existence of a suboptimal expert, such as a heuristic, a model-based controller, or a policy trained on a related task. Such an expert is often available and can guide early training, but its sub-optimality limits final performance. The challenge then becomes balancing expert guidance against learning from rewards. Existing methods set the expert's influence through a blending weight, a schedule, or an evaluation-driven curriculum. Alternatively, they adapt it with additional learned components such as critics over expert actions or auxiliary agents. However, none optimizes it using the same on-policy objective as the policy itself. We propose a method in which the learner and the expert alternate control within each training episode, and the expert's share of control is a single learnable parameter optimized jointly with the policy. The learner benefits from the expert early in training, but its share of control declines as the learner becomes more competent, until eventually vanishing completely. This leaves the learner acting alone and better than the suboptimal expert. We evaluate our method on 34 tasks across two benchmarks, spanning discrete and continuous action spaces, using both learned and model-based experts. Our method improves sample efficiency over guided and unguided baselines while requiring minimal hyperparameter variation. The expert's share decays to zero as the learner improves, vanishing when the expert is no longer useful.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 33 of its 46 references that are in the map. First posted on arXiv 5 Oct 2026.

Builds on

33 of 46 references.

References · 33

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  3. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  4. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  5. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  6. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  7. 2016The Option-Critic Architecturecited by 448
  8. 2015Policy Distillationcited by 440
  9. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  10. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  11. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  12. 2025Learning to Reason under Off-Policy Guidancecited by 281
21 more
  1. 2018Residual Reinforcement Learning for Robot Controlcited by 274
  2. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 252
  3. 2018Residual Policy Learningcited by 164
  4. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 104
  5. 2023PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNavcited by 93
  6. 2025MuJoCo Playgroundcited by 78
  7. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 65
  8. 2019MPC-Net: A First Principles Guided Policy Searchcited by 34
  9. 2020Blending MPC & Value Function Approximation for Efficient Reinforcement Learningcited by 32
  10. 2019AC-Teach: A Bayesian Actor-Critic Method for Policy Learning with an Ensemble of Suboptimal Teacherscited by 28
  11. 2018Fast Policy Learning through Imitation and Reinforcementcited by 27
  12. 2021Bayesian Controller Fusion: Leveraging Control Priors in Deep Reinforcement Learning for Roboticscited by 20
  13. 2023Actor-Critic Model Predictive Control: Differentiable Optimization meets Reinforcement Learning for Agile Flightcited by 19
  14. 2024NAVIX: Scaling MiniGrid Environments with JAXcited by 18
  15. 2018Learning with Training Wheels: Speeding up Training with a Simple Controller for Deep Reinforcement Learningcited by 17
  16. 2023TGRL: An Algorithm for Teacher Guided Reinforcement Learningcited by 17
  17. 2023Blending Imitation and Reinforcement Learning for Robust Policy Improvementcited by 17
  18. 2022A Hierarchical Approach for Strategic Motion Planning in Autonomous Racingcited by 7
  19. 2023Reinforcement Learning by Guided Safe Explorationcited by 7
  20. 2025Differentiable Model Predictive Control on the GPUcited by 7
  21. 2025Residual MPC: Blending Reinforcement Learning with GPU-Parallelized Model Predictive Controlcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Trust the Batch, On- or Off-Policy: Adaptive Policy Optimization for RL Post-Training
  2. 2025Guided Policy Optimization under Partial Observability
  3. 2024Efficient Offline Reinforcement Learning: First Imitate, then Improve
  4. 2026Rationale-Guided Policy Optimization: Learning to Reason with Adaptive Rationale Scaffolding
  5. 2026An Agency-Transferring Model-Free Policy Enhancement Technique
  6. 2024Enhancing Sample Efficiency and Exploration in Reinforcement Learning through the Integration of Diffusion Models and Proximal Policy Optimization
  7. 2023Hundreds Guide Millions: Adaptive Offline Reinforcement Learning with Expert Guidance
  8. 2026KLip-PPO: A per-sample KL perspective on PPO-Clip