mlsanityData to 7 Oct 2026

Guarded Policy Optimization with Imperfect Online Demonstrations

arXiv 2303.01728 ↗First posted 3 Mar 2023cs.LG cs.AI cs.ROcited by 10

Not on the map. Why?

Abstract (arXiv)

The Teacher-Student Framework (TSF) is a reinforcement learning setting where a teacher agent guards the training of a student agent by intervening and providing online demonstrations. Assuming optimal, the teacher policy has the perfect timing and capability to intervene in the learning process of the student agent, providing safety guarantee and exploration guidance. Nevertheless, in many real-world settings it is expensive or even impossible to obtain a well-performing teacher policy. In this work, we relax the assumption of a well-performing teacher and develop a new method that can incorporate arbitrary teacher policies with modest or inferior performance. We instantiate an Off-Policy Reinforcement Learning algorithm, termed Teacher-Student Shared Control (TS2C), which incorporates teacher intervention based on trajectory-based value estimation. Theoretical analysis validates that the proposed TS2C algorithm attains efficient exploration and substantial safety guarantee without being affected by the teacher's own performance. Experiments on various continuous control tasks show that our method can exploit teacher policies at different performance levels while maintaining a low training cost. Moreover, the student policy surpasses the imperfect teacher policy in terms of higher accumulated reward in held-out testing environments. Code is available at https://metadriverse.github.io/TS2C.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 35 of its 46 references that are in the map. First posted on arXiv 3 Mar 2023.

Builds on

35 of 46 references.

References · 35

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  6. 2016Generative Adversarial Imitation Learningcited by 1,576
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  9. 2017Constrained Policy Optimizationcited by 928
  10. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  11. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  12. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
23 more
  1. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  2. 2015Policy Distillationcited by 414
  3. 2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learningcited by 281
  4. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  5. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 245
  6. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 116
  7. 2020Human-in-the-Loop Imitation Learning using Remote Teleoperationcited by 103
  8. 2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
  9. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  10. 2019A survey on intrinsic motivation in reinforcement learningcited by 79
  11. 2018Kickstarting Deep Reinforcement Learningcited by 75
  12. 2021Safe Reinforcement Learning Using Advantage-Based Interventioncited by 45
  13. 2021Confidence-Aware Imitation Learning from Demonstrations with Varying Optimalitycited by 44
  14. 2022Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimizationcited by 44
  15. 2019DisCoRL: Continual Reinforcement Learning via Policy Distillationcited by 35
  16. 2021Safe Driving via Expert Guided Policy Optimizationcited by 29
  17. 2021Correct Me if I am Wrong: Interactive Learning for Robotic Manipulationcited by 24
  18. 2017Agent-Agnostic Human-in-the-Loop Reinforcement Learningcited by 23
  19. 2020Non-local Policy Optimization via Diversity-regularized Collaborative Explorationcited by 13
  20. 2020Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillationcited by 10
  21. 2017Collaborative Deep Reinforcement Learningcited by 5
  22. 2019On Value Discrepancy of Imitation Learningcited by 4
  23. 2022ReIL: A Framework for Reinforced Intervention-based Imitation Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 4; 2026: 1.

10 papers.

  1. 2023State Regularized Policy Optimization on Data with Dynamics Shiftcited by 23
  2. 2023Residual Q-Learning: Offline and Online Policy Customization without Valuecited by 9
  3. 2024Trustworthy Human-AI Collaboration: Reinforcement Learning with Human Feedback and Physics Knowledge for Safe Autonomous Drivingcited by 8
  4. 2024Knowledge Transfer from Simple to Complex: A Safe and Efficient Reinforcement Learning Framework for Autonomous Driving Decision-Makingcited by 5
  5. 2024BeTAIL: Behavior Transformer Adversarial Imitation Learning from Human Racing Gameplaycited by 3
  6. 2025Data-Efficient Learning from Human Interventions for Mobile Robotscited by 3
  7. 2025Predictive Preference Learning from Human Interventionscited by 3
  8. 2025Policy Regularization on Globally Accessible States in Cross-Dynamics Reinforcement Learningcited by 0
  9. 2025Confidence-Guided Human-AI Collaboration: Reinforcement Learning with Distributional Proxy Value Propagation for Autonomous Drivingcited by 0
  10. 2026Uncertainty-Aware and Temporally Regulated Expert Advice in Reinforcement Learning for Autonomous Drivingcited by 0