mlsanityData to 7 Oct 2026

Solving Compositional Reinforcement Learning Problems via Task Reduction

arXiv 2103.07607 ↗First posted 13 Mar 2021cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

We propose a novel learning paradigm, Self-Imitation via Reduction (SIR), for solving compositional reinforcement learning problems. SIR is based on two core ideas: task reduction and self-imitation. Task reduction tackles a hard-to-solve task by actively reducing it to an easier task whose solution is known by the RL agent. Once the original hard task is successfully solved by task reduction, the agent naturally obtains a self-generated solution trajectory to imitate. By continuously collecting and imitating such demonstrations, the agent is able to progressively expand the solved subspace in the entire task space. Experiment results show that SIR can significantly accelerate and improve learning on a variety of challenging sparse-reward continuous-control problems with compositional structures. Code and videos are available at https://sites.google.com/view/sir-compositional.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 50 of its 81 references that are in the map. First posted on arXiv 13 Mar 2021.

Builds on

50 of 81 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2016Generative Adversarial Imitation Learningcited by 1,576
  6. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  7. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  8. 2017Hindsight Experience Replaycited by 1,096
  9. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  10. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  11. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  12. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
38 more
  1. 2016The Option-Critic Architecturecited by 433
  2. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 351
  3. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  4. 2019Learning Latent Plans from Playcited by 310
  5. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
  6. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 282
  7. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  8. 2017Automated Curriculum Learning for Neural Networkscited by 251
  9. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  10. 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 202
  11. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 200
  12. 2017Teacher-Student Curriculum Learningcited by 197
  13. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  14. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
  15. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
  16. 2019Goal-conditioned Imitation Learningcited by 159
  17. 2017DARLA: Improving Zero-Shot Transfer in Reinforcement Learningcited by 147
  18. 2019Planning with Goal-Conditioned Policiescited by 144
  19. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  20. 2018Self-Imitation Learningcited by 135
  21. 2017Stochastic Neural Networks for Hierarchical Reinforcement Learningcited by 124
  22. 2018Zero-Shot Visual Imitationcited by 114
  23. 2019Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environmentscited by 113
  24. 2019MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policiescited by 105
  25. 2018Unsupervised Control Through Non-Parametric Discriminative Rewardscited by 102
  26. 2016A Deep Hierarchical Approach to Lifelong Learning in Minecraftcited by 90
  27. 2014Freeze-Thaw Bayesian Optimizationcited by 86
  28. 2019Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generationcited by 83
  29. 2018Composable Deep Reinforcement Learning for Robotic Manipulationcited by 70
  30. 2019Towards Practical Multi-Object Manipulation using Relational Reinforcement Learningcited by 54
  31. 2019Exploration via Hindsight Goal Generationcited by 53
  32. 2019Contextual Imagined Goals for Self-Supervised Robotic Learningcited by 40
  33. 2018Automatically Composing Representation Transformations as a Means for Generalizationcited by 35
  34. 2020Data-efficient Hindsight Off-policy Option Learningcited by 31
  35. 2019Learning to Reach Goals via Iterated Supervised Learningcited by 26
  36. 2017Learning to Compose Skillscited by 15
  37. 2018Floyd-Warshall Reinforcement Learning: Learning from Past Experiences to Reach New Goalscited by 8
  38. 2019Memory Based Trajectory-conditioned Policies for Learning from Sparse Rewardscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 6; 2023: 3; 2024: 3; 2025: 1; 2026: 2.

19 papers.

Sort:

By area:

  1. 2022Environment Generation for Zero-Shot Compositional Reinforcement Learningcited by 32
  2. 2022How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Compositioncited by 27
  3. 2022Phasic Self-Imitative Reduction for Sparse-Reward Goal-Conditioned Reinforcement Learningcited by 16
  4. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
  5. 2022A System for Morphology-Task Generalization via Unified Representation and Behavior Distillationcited by 12
  6. 2025Transitive RL: Value Learning via Divide and Conquercited by 12
  7. 2021Multi-Task Learning with Sequence-Conditioned Transporter Networkscited by 8
  8. 2021Learning to Design and Construct Bridge without Blueprintcited by 6
  9. 2022E-MAPP: Efficient Multi-Agent Reinforcement Learning with Parallel Program Guidancecited by 6
  10. 2022Categorical semantics of compositional reinforcement learningcited by 5
  11. 2023Camouflaged Image Synthesis Is All You Need to Boost Camouflaged Detectioncited by 5
  12. 2024A General Theory for Compositional Generalizationcited by 4
7 more
  1. 2021DAIR: Disentangled Attention Intrinsic Regularization for Safe and Efficient Bimanual Manipulationcited by 3
  2. 2021A Simple Approach to Continual Learning by Transferring Skill Parameterscited by 2
  3. 2024Reduce, Reuse, Recycle: Categories for Compositional Reinforcement Learningcited by 2
  4. 2023Towards Zero-Shot Functional Compositionality of Language Modelscited by 1
  5. 2024MeMo: Meaningful, Modular Controllers via Noise Injectioncited by 1
  6. 2026Temporal Self-Imitation Learningcited by 0
  7. 2026GTRL: Grounding Divide-and-Conquer Value Learning with Temporal Differencescited by 0