mlsanityData to 7 Oct 2026

Adversarial Intrinsic Motivation for Reinforcement Learning

arXiv 2105.13345 ↗First posted 27 May 2021cs.LGcited by 40

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Learning with an objective to minimize the mismatch with a reference distribution has been shown to be useful for generative modeling and imitation learning. In this paper, we investigate whether one such objective, the Wasserstein-1 distance between a policy's state visitation distribution and a target distribution, can be utilized effectively for reinforcement learning (RL) tasks. Specifically, this paper focuses on goal-conditioned reinforcement learning where the idealized (unachievable) target distribution has full measure at the goal. This paper introduces a quasimetric specific to Markov Decision Processes (MDPs) and uses this quasimetric to estimate the above Wasserstein-1 distance. It further shows that the policy that minimizes this Wasserstein-1 distance is the policy that reaches the goal in as few steps as possible. Our approach, termed Adversarial Intrinsic Motivation (AIM), estimates this Wasserstein-1 distance through its dual objective and uses it to compute a supplemental reward function. Our experiments show that this reward function changes smoothly with respect to transitions in the MDP and directs the agent's exploration to find the goal efficiently. Additionally, we combine AIM with Hindsight Experience Replay (HER) and show that the resulting algorithm accelerates learning significantly on several simulated robotics tasks when compared to other rewards that encourage exploration or accelerate learning.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 39 of its 83 references that are in the map. First posted on arXiv 27 May 2021.

Builds on

39 of 83 references.

Closest ancestors · 5

Same direction, newest first

  1. 2020Primal Wasserstein Imitation Learning8 Jun
  2. 2019A Divergence Minimization Perspective on Imitation Learning Methods6 Nov
  3. 2019Wasserstein Adversarial Imitation Learning19 Jun
  4. 2019Goal-conditioned Imitation Learning13 Jun
  5. 2018Generative Adversarial Imitation from Observation17 Jul

Other references · 34

  1. 2015Continuous control with deep reinforcement learningcited by 3,279
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2017Improved Training of Wasserstein GANscited by 2,223
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  6. 2017Hindsight Experience Replaycited by 1,096
  7. 2018Exploration by Random Network Distillationcited by 966
  8. 2018Computational Optimal Transportcited by 765
  9. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  10. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  11. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  12. 2018Provably Efficient Maximum Entropy Explorationcited by 240
22 more
  1. 2019Efficient Exploration via State Marginal Matchingcited by 186
  2. 2018RUDDER: Return Decomposition for Delayed Rewardscited by 162
  3. 2017The Cramer Distance as a Solution to Biased Wasserstein Gradientscited by 116
  4. 2021Reward (Mis)design for Autonomous Drivingcited by 102
  5. 2018On Learning Intrinsic Rewards for Policy Gradient Methodscited by 100
  6. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 99
  7. 2020Automatic Curriculum Learning through Value Disagreementcited by 86
  8. 2017Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learningcited by 79
  9. 2018Synthesizing Programs for Images using Reinforced Adversarial Learningcited by 72
  10. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matchingcited by 67
  11. 2015Deep Reinforcement Learning in Parameterized Action Spacecited by 65
  12. 2019Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discoverycited by 65
  13. 2021Adversarially Guided Actor-Criticcited by 47
  14. 2017From optimal transport to generative modeling: the VEGAN cookbookcited by 41
  15. 2019Is the Policy Gradient a Gradient?cited by 40
  16. 2021Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classificationcited by 38
  17. 2019What Can Learned Intrinsic Rewards Capture?cited by 36
  18. 2021Wasserstein GANs Work Because They Fail (to Approximate the Wasserstein Distance)cited by 33
  19. 2021An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learningcited by 20
  20. 2019Self-supervised Learning of Distance Functions for Goal-Conditioned Reinforcement Learningcited by 18
  21. 2021DisCo RL: Distribution-Conditioned Reinforcement Learning for General-Purpose Policiescited by 9
  22. 2017The Stochastic Shortest Path Problem : A polyhedral combinatorics perspectivecited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 4; 2023: 13; 2024: 7; 2025: 7; 2026: 8.

40 papers.

Sort:

By area:

  1. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  2. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  3. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
  4. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  5. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 65
  6. 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Makingcited by 53
  7. 2023Contrastive Difference Predictive Codingcited by 35
  8. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
  9. 2024Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodingscited by 31
  10. 2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learningcited by 22
  11. 2023Fantastic Rewards and How to Tame Them: A Case Study on Reward Learning for Task-oriented Dialogue Systemscited by 19
  12. 2025Episodic Novelty Through Temporal Distancecited by 15
28 more
  1. 2023Outcome-directed Reinforcement Learning by Uncertainty & Temporal Distance-Aware Curriculum Goal Generationcited by 14
  2. 2023f-Policy Gradients: A General Framework for Goal Conditioned RL using f-Divergencescited by 14
  3. 2025DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learningcited by 11
  4. 2025State-Covering Trajectory Stitching for Diffusion Plannerscited by 11
  5. 2024Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNetscited by 8
  6. 2023Imitation Learning from Observation through Optimal Transportcited by 7
  7. 2024RLZero: Direct Policy Inference from Language Without In-Domain Supervisioncited by 7
  8. 2023On the Benefit of Optimal Transport for Curriculum Reinforcement Learningcited by 6
  9. 2025Goal Reaching with Eikonal-Constrained Hierarchical Quasimetric Reinforcement Learningcited by 6
  10. 2021Wasserstein Distance Maximizing Intrinsic Controlcited by 5
  11. 2023Goal-conditioned Offline Reinforcement Learning through State Space Partitioningcited by 3
  12. 2024Goal-Conditioned Reinforcement Learning from Sub-Optimal Data on Metric Spacescited by 3
  13. 2024Zero-Shot Offline Imitation Learning via Optimal Transportcited by 3
  14. 2025Efficient Skill Discovery via Regret-Aware Optimizationcited by 3
  15. 2022Task Phasing: Automated Curriculum Learning from Demonstrationscited by 2
  16. 2023Magnetic Field-Based Reward Shaping for Goal-Conditioned Reinforcement Learningcited by 2
  17. 2024Quasimetric Value Functions with Dense Rewardscited by 2
  18. 2026Action-Sufficient Goal Representationscited by 1
  19. 2026Occupancy Reward Shaping: Improving Credit Assignment for Offline Goal-Conditioned Reinforcement Learningcited by 1
  20. 2026Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learningcited by 1
  21. 2026COLLIE: Guiding Skill Discovery in Semantically Coherent Latent Spacecited by 1
  22. 2022Understanding Hindsight Goal Relabeling from a Divergence Minimization Perspectivecited by 0
  23. 2023MRHER: Model-based Relay Hindsight Experience Replay for Sequential Object Manipulation Tasks with Sparse Rewardscited by 0
  24. 2025Inter-Agent Relative Representations for Multi-Agent Option Discoverycited by 0
  25. 2026Group-Invariant Unsupervised Skill Discovery: Symmetry-aware Skill Representations for Generalizable Behaviorcited by 0
  26. 2026Physics-informed Goal-Conditioned Reinforcement Learning under Hybrid Contact Dynamicscited by 0
  27. 2026Learning Process Rewards via Success Visitation Matching for Efficient RLcited by 0
  28. 2026Improving Offline Goal-Conditioned Reinforcement Learning via Selective Reward Stimulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Wasserstein Distance guided Adversarial Imitation Learning with Reward Shape Exploration
  2. 2020Nested-Wasserstein Self-Imitation Learning for Sequence Generation
  3. 2022Towards Improving Exploration in Self-Imitation Learning using Intrinsic Motivation
  4. 2023Generative Intrinsic Optimization: Intrinsic Control with Model Learning
  5. 2018Learning Self-Imitating Diverse Policies
  6. 2023Intrinsic Motivation in Model-based Reinforcement Learning: A Brief Review
  7. 2025Offline Goal-conditioned Reinforcement Learning with Quasimetric Representations
  8. 2024The intrinsic motivation of reinforcement and imitation learning for sequential tasks