mlsanityData to 7 Oct 2026

Asymmetric self-play for automatic goal discovery in robotic manipulation

arXiv 2101.04882 ↗First posted 13 Jan 2021cs.LG cs.AI cs.CVcited by 71

Not on the map. Why?

Abstract (arXiv)

We train a single, goal-conditioned policy that can solve many robotic manipulation tasks, including tasks with previously unseen goals and objects. We rely on asymmetric self-play for goal discovery, where two agents, Alice and Bob, play a game. Alice is asked to propose challenging goals and Bob aims to solve them. We show that this method can discover highly diverse and complex goals without any human priors. Bob can be trained with only sparse rewards, because the interaction between Alice and Bob results in a natural curriculum and Bob can learn from Alice's trajectory when relabeled as a goal-conditioned demonstration. Finally, our method scales, resulting in a single policy that can generalize to many unseen tasks such as setting a table, stacking blocks, and solving simple puzzles. Videos of a learned policy is available at https://robotics-self-play.github.io.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Builds on 47 of its 59 references that are in the map. First posted on arXiv 13 Jan 2021.

Builds on

47 of 59 references.

References · 47

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  4. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  6. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  7. 2017Hindsight Experience Replaycited by 1,096
  8. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  9. 2018Exploration by Random Network Distillationcited by 966
  10. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  11. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  12. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
35 more
  1. 2019Solving Rubik's Cube with a Robot Handcited by 699
  2. 2018Learning Dexterous In-Hand Manipulationcited by 643
  3. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  4. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  5. 2019Learning agile and dynamic motor skills for legged robotscited by 402
  6. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  7. 2016Learning to reinforcement learncited by 369
  8. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  9. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  10. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  11. 2019Learning Latent Plans from Playcited by 310
  12. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  13. 2020First return, then explorecited by 256
  14. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  15. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  16. 2017One-Shot Imitation Learningcited by 232
  17. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
  18. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 207
  19. 2017Teacher-Student Curriculum Learningcited by 197
  20. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 185
  21. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 182
  22. 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 171
  23. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
  24. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 139
  25. 2018Self-Imitation Learningcited by 135
  26. 2017Emergent Complexity via Multi-Agent Competitioncited by 117
  27. 2020Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutionscited by 103
  28. 2016Sim-to-Real Robot Learning from Pixels with Progressive Netscited by 89
  29. 2020Automatic Curriculum Learning through Value Disagreementcited by 86
  30. 2018Learning Montezuma's Revenge from a Single Demonstrationcited by 76
  31. 2017Data-efficient Deep Reinforcement Learning for Dexterous Manipulationcited by 61
  32. 2013Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robotscited by 56
  33. 2019Towards Practical Multi-Object Manipulation using Relational Reinforcement Learningcited by 54
  34. 2018Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learningcited by 23
  35. 2019Competitive Experience Replaycited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 19; 2023: 13; 2024: 7; 2025: 15; 2026: 8.

71 papers.

Sort:

By area:

  1. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
  2. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  3. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
  4. 2022Evolving Curricula with Regret-Based Environment Designcited by 166
  5. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 158
  6. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  7. 2021Discovering and Achieving Goals via World Modelscited by 131
  8. 2021Replay-Guided Adversarial Environment Designcited by 108
  9. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  10. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  11. 2023OMNI: Open-endedness via Models of human Notions of Interestingnesscited by 66
  12. 2025Robust Autonomy Emerges from Self-Playcited by 64
38 more
  1. 2025Self-Challenging Language Model Agentscited by 56
  2. 2022Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulationcited by 49
  3. 2023Controllability-Aware Unsupervised Skill Discoverycited by 49
  4. 2025Self-Questioning Language Modelscited by 47
  5. 2025Language Self-Play For Data-Free Trainingcited by 42
  6. 2026Dr. Zero: Self-Evolving Search Agents without Training Datacited by 39
  7. 2023Decomposing the Enigma: Subgoal-based Demonstration Learning for Formal Theorem Provingcited by 37
  8. 2021Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraftcited by 34
  9. 2022Curious Exploration via Structured World Models Yields Zero-Shot Object Manipulationcited by 29
  10. 2023Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shapingcited by 25
  11. 2022Blocks Assemble! Learning to Assemble with Large-Scale Structured Reinforcement Learningcited by 21
  12. 2024A social path to human-like artificial intelligencecited by 20
  13. 2024A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoalscited by 20
  14. 2022Emergent Bartering Behaviour in Multi-Agent Reinforcement Learningcited by 18
  15. 2022DualAfford: Learning Collaborative Visual Affordance for Dual-gripper Manipulationcited by 18
  16. 2023Stabilizing Unsupervised Environment Design with a Learned Adversarycited by 17
  17. 2022Grounding Aleatoric Uncertainty for Unsupervised Environment Designcited by 16
  18. 2022Understanding the Complexity Gains of Single-Task RL with a Curriculumcited by 16
  19. 2021Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Explorationcited by 14
  20. 2022It Takes Four to Tango: Multiagent Selfplay for Automatic Curriculum Generationcited by 14
  21. 2023f-Policy Gradients: A General Framework for Goal Conditioned RL using f-Divergencescited by 14
  22. 2025Normalizing Flows are Capable Models for Continuous Controlcited by 13
  23. 2026Teaching Models to Teach Themselves: Reasoning at the Edge of Learnabilitycited by 13
  24. 2024Learning to Drive via Asymmetric Self-Playcited by 11
  25. 2025Automated Skill Discovery for Language Agents through Exploration and Iterative Feedbackcited by 11
  26. 2025Learning to Generate Unit Test via Adversarial Reinforcement Learningcited by 11
  27. 2025SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Modelscited by 9
  28. 2026π-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Datacited by 7
  29. 2021Explore and Control with Adversarial Surprisecited by 6
  30. 2023Regularity as Intrinsic Reward for Free Playcited by 5
  31. 2025KnowRL: Teaching Language Models to Know What They Knowcited by 5
  32. 2022Feasible Adversarial Robust Reinforcement Learning for Underspecified Environmentscited by 4
  33. 2025BuilderBench: The Building Blocks of Intelligent Agentscited by 4
  34. 2026GASP: Guided Asymmetric Self-Play For Coding LLMscited by 4
  35. 2021Cogment: Open Source Framework For Distributed Multi-actor Training, Deployment & Operationscited by 3
  36. 2022Walk the Random Walk: Learning to Discover and Reach Goals Without Supervisioncited by 3
  37. 2023Stackelberg Games for Learning Emergent Behaviors During Competitive Autocurriculacited by 3
  38. 2024Unsupervised Skill Discovery for Robotic Manipulation through Automatic Task Generationcited by 3

50 most cited of 71 All 71 (JSON)