mlsanityData to 7 Oct 2026

Robot Policy Learning from Demonstration Using Advantage Weighting and Early Termination

arXiv 2208.00478 ↗First posted 31 Jul 2022cs.LG cs.ROcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Learning robotic tasks in the real world is still highly challenging and effective practical solutions remain to be found. Traditional methods used in this area are imitation learning and reinforcement learning, but they both have limitations when applied to real robots. Combining reinforcement learning with pre-collected demonstrations is a promising approach that can help in learning control policies to solve robotic tasks. In this paper, we propose an algorithm that uses novel techniques to leverage offline expert data using offline and online training to obtain faster convergence and improved performance. The proposed algorithm (AWET) weights the critic losses with a novel agent advantage weight to improve over the expert data. In addition, AWET makes use of an automatic early termination technique to stop and discard policy rollouts that are not similar to expert trajectories -- to prevent drifting far from the expert data. In an ablation study, AWET showed improved and promising performance when compared to state-of-the-art baselines on four standard robotic tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 38 of its 65 references that are in the map. First posted on arXiv 31 Jul 2022.

Builds on

38 of 65 references.

Closest ancestors · 6

Same direction, newest first

  1. 2020Critic Regularized Regression26 Jun
  2. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  5. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  6. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 32

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2016OpenAI Gymcited by 1,684
  7. 2016Generative Adversarial Imitation Learningcited by 1,576
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  9. 2017Hindsight Experience Replaycited by 1,096
  10. 2015Prioritized Experience Replaycited by 1,070
  11. 2016End to End Learning for Self-Driving Carscited by 1,000
  12. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
20 more
  1. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  2. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  3. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  4. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  5. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  6. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  7. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  8. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  9. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  10. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  11. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  12. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  13. 2018Residual Reinforcement Learning for Robot Controlcited by 242
  14. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  15. 2018DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skillscited by 153
  16. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrationscited by 114
  17. 2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
  18. 2018Reinforcement Learning from Imperfect Demonstrationscited by 74
  19. 2021A Study on Dense and Sparse (Visual) Rewards in Robot Policy Learningcited by 3
  20. 2021Neural Task Success Classifiers for Robotic Manipulation from Few Real Demonstrationscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Robot Policy Transfer with Online Demonstrations: An Active Reinforcement Learning Approach
  2. 2022Latent-Variable Advantage-Weighted Policy Optimization for Offline RL
  3. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning
  4. 2026RoboFFT: Finetuning generative robot policy via online reinforcement learning with forward process ↗
  5. 2024Efficient Offline Reinforcement Learning: First Imitate, then Improve
  6. 2023GO-DICE: Goal-Conditioned Option-Aware Offline Imitation Learning via Stationary Distribution Correction Estimation
  7. 2022Offline Robot Reinforcement Learning with Uncertainty-Guided Human Expert Sampling
  8. 2021AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scale