mlsanityData to 14 Oct 2026

Match or Replay: Self Imitating Proximal Policy Optimization

arXiv 2603.27515 ↗First posted 29 Mar 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Reinforcement Learning (RL) agents often struggle with inefficient exploration, particularly in environments with sparse rewards. Traditional exploration strategies can lead to slow learning and suboptimal performance because agents fail to systematically build on previously successful experiences, thereby reducing sample efficiency. To tackle this issue, we propose a self-imitating on-policy algorithm that enhances exploration and sample efficiency by leveraging past high-reward state-action pairs to guide policy updates. Our method incorporates self-imitation by using optimal transport distance in dense reward environments to prioritize state visitation distributions that match the most rewarding trajectory. In sparse-reward environments, we uniformly replay successful self-encountered trajectories to facilitate structured exploration. Experimental results across diverse environments demonstrate substantial improvements in learning efficiency, including MuJoCo for dense rewards and the partially observable 3D Animal-AI Olympics and multi-goal PointMaze for sparse rewards. Our approach achieves faster convergence and significantly higher success rates compared to state-of-the-art self-imitating RL baselines. These findings underscore the potential of self-imitation as a robust strategy for enhancing exploration in RL, with applicability to more complex tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 36 of its 47 references that are in the map. First posted on arXiv 29 Mar 2026.

Builds on

36 of 47 references.

References · 36

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  4. 2015Prioritized Experience Replaycited by 1,099
  5. 2018Exploration by Random Network Distillationcited by 993
  6. 2018Computational Optimal Transportcited by 780
  7. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  8. 2017Deep Reinforcement Learning: An Overviewcited by 333
  9. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 313
  10. 2017One-Shot Imitation Learningcited by 233
  11. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 212
  12. 2018Self-Imitation Learningcited by 138
24 more
  1. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 134
  2. 2022Jump-Start Reinforcement Learningcited by 112
  3. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 82
  4. 2018Learning Montezuma's Revenge from a Single Demonstrationcited by 80
  5. 2022Watch and Match: Supercharging Imitation with Regularized Optimal Transportcited by 67
  6. 2023Optimal Transport for Offline Imitation Learningcited by 42
  7. 2021Generalized Proximal Policy Optimization with Sample Reusecited by 40
  8. 2018Deep Drone Racing: Learning Agile Flight in Dynamic Environmentscited by 34
  9. 2018Learning Self-Imitating Diverse Policiescited by 29
  10. 2019Watch, Try, Learn: Meta-Learning from Demonstrations and Rewardcited by 28
  11. 2021Self-Imitation Learning by Planningcited by 26
  12. 2018Generative Adversarial Self-Imitation Learningcited by 25
  13. 2023Improved Sample Complexity Analysis of Natural Policy Gradient Algorithm with General Parameterization for Infinite Horizon Discounted Reward Markov Decision Processescited by 25
  14. 2023Demonstration-Guided Reinforcement Learning with Efficient Exploration for Task Automation of Surgical Robotcited by 23
  15. 2020Guided Exploration with Proximal Policy Optimization using a Single Demonstrationcited by 19
  16. 2020Self-Imitation Advantage Learningcited by 15
  17. 2024How to Leverage Demonstration Data in Alignment for Large Language Model? A Self-Imitation Learning Perspectivecited by 14
  18. 2020Self-Imitation Learning via Generalized Lower Bound Q-learningcited by 13
  19. 2023Cooperating Graph Neural Networks with Deep Reinforcement Learning for Vaccine Prioritizationcited by 9
  20. 2018Deep RTS: A Game Environment for Deep Reinforcement Learning in Real-Time Strategy Gamescited by 7
  21. 2023Visual Hindsight Self-Imitation Learning for Interactive Navigationcited by 6
  22. 2023Reinforcement Learning in Robotic Motion Planning by Combined Experience-based Planning and Self-Imitation Learningcited by 4
  23. 2025MOORL: A Framework for Integrating Offline-Online Reinforcement Learningcited by 4
  24. 2025From Novelty to Imitation: Self-Distilled Rewards for Offline Reinforcement Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Enhanced Generalization through Prioritization and Diversity in Self-Imitation Reinforcement Learning over Procedural Environments with Sparse Rewards
  2. 2022Towards Improving Exploration in Self-Imitation Learning using Intrinsic Motivation
  3. 2025Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning
  4. 2021Learning from Guided Play: A Scheduled Hierarchical Approach for Improving Exploration in Adversarial Imitation Learning
  5. 2019Reinforced Imitation in Heterogeneous Action Space
  6. 2026Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying
  7. 2025Beyond-Expert Performance with Limited Demonstrations: Efficient Imitation Learning with Double Exploration
  8. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training