mlsanityData to 14 Oct 2026

Phasic Self-Imitative Reduction for Sparse-Reward Goal-Conditioned Reinforcement Learning

arXiv 2206.12030 ↗First posted 24 Jun 2022cs.LG cs.AIcited by 16

Not on the map. Why?

Abstract (arXiv)

It has been a recent trend to leverage the power of supervised learning (SL) towards more effective reinforcement learning (RL) methods. We propose a novel phasic approach by alternating online RL and offline SL for tackling sparse-reward goal-conditioned problems. In the online phase, we perform RL training and collect rollout data while in the offline phase, we perform SL on those successful trajectories from the dataset. To further improve sample efficiency, we adopt additional techniques in the online phase including task reduction to generate more feasible trajectories and a value-difference-based intrinsic reward to alleviate the sparse-reward issue. We call this overall algorithm, PhAsic self-Imitative Reduction (PAIR). PAIR substantially outperforms both non-phasic RL and phasic SL baselines on sparse-reward goal-conditioned robotic control problems, including a challenging stacking task. PAIR is the first RL method that learns to stack 6 cubes with only 0/1 success rewards from scratch.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 50 of its 67 references that are in the map. First posted on arXiv 24 Jun 2022.

Builds on

50 of 67 references.

References · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2015Continuous control with deep reinforcement learningcited by 3,407
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  9. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  10. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  11. 2017Hindsight Experience Replaycited by 1,136
  12. 2018Exploration by Random Network Distillationcited by 993
38 more
  1. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  2. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  3. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  4. 2019Solving Rubik's Cube with a Robot Handcited by 734
  5. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  6. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  7. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  8. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  9. 2014Convex Optimization: Algorithms and Complexitycited by 575
  10. 2019Learning agile and dynamic motor skills for legged robotscited by 504
  11. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  12. 2016The Option-Critic Architecturecited by 448
  13. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  14. 2019Learning Latent Plans from Playcited by 316
  15. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
  16. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  17. 2020First return, then explorecited by 266
  18. 2018Visual Reinforcement Learning with Imagined Goalscited by 251
  19. 2020Never Give Up: Learning Directed Exploration Strategiescited by 212
  20. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 185
  21. 2019Planning with Goal-Conditioned Policiescited by 149
  22. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 125
  23. 2020Phasic Policy Gradientcited by 123
  24. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  25. 2022Jump-Start Reinforcement Learningcited by 112
  26. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 106
  27. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 103
  28. 2019Reward-Conditioned Policiescited by 102
  29. 2018Temporal Difference Models: Model-Free Deep RL for Model-Based Controlcited by 89
  30. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  31. 2019Towards Practical Multi-Object Manipulation using Relational Reinforcement Learningcited by 55
  32. 2018The Mirage of Action-Dependent Baselines in Reinforcement Learningcited by 48
  33. 2021AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scalecited by 44
  34. 2019Maximum Entropy-Regularized Multi-Goal Reinforcement Learningcited by 40
  35. 2021Self-Imitation Learning by Planningcited by 26
  36. 2018Many-Goals Reinforcement Learningcited by 22
  37. 2021Solving Compositional Reinforcement Learning Problems via Task Reductioncited by 19
  38. 2021Understanding the World Through Actioncited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 8; 2024: 4; 2025: 0; 2026: 1.

16 papers.

Sort:

By area:

  1. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  2. 2023Decomposing the Enigma: Subgoal-based Demonstration Learning for Formal Theorem Provingcited by 37
  3. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 36
  4. 2023Universal Visual Decomposer: Long-Horizon Manipulation Made Easycited by 25
  5. 2024DecisionNCE: Embodied Multimodal Representations via Implicit Preference Learningcited by 20
  6. 2022A System for Morphology-Task Generalization via Unified Representation and Behavior Distillationcited by 13
  7. 2023DexCatch: Learning to Catch Arbitrary Objects with Dexterous Handscited by 8
  8. 2024On-Robot Reinforcement Learning with Goal-Contrastive Rewardscited by 8
  9. 2023Goal-conditioned Offline Reinforcement Learning through State Space Partitioningcited by 3
  10. 2023When Learning Is Out of Reach, Reset: Generalization in Autonomous Visuomotor Reinforcement Learningcited by 3
  11. 2023User-centric Heterogeneous-action Deep Reinforcement Learning for Virtual Reality in the Metaverse over Wireless Networkscited by 1
  12. 2023SEGO: Sequential Subgoal Optimization for Mathematical Problem-Solvingcited by 1
4 more
  1. 2023Heterogeneous 360 Degree Videos in Metaverse: Differentiated Reinforcement Learning Approachescited by 0
  2. 2024Spatio-temporal Value Semantics-based Abstraction for Dense Deep Reinforcement Learningcited by 0
  3. 2024Improving Reward-Conditioned Policies for Multi-Armed Bandits using Normalized Weight Functionscited by 0
  4. 2026Temporal Self-Imitation Learningcited by 0