mlsanityData to 7 Oct 2026

Robust Imitation via Mirror Descent Inverse Reinforcement Learning

arXiv 2210.11201 ↗First posted 20 Oct 2022cs.LGcited by 5

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Recently, adversarial imitation learning has shown a scalable reward acquisition method for inverse reinforcement learning (IRL) problems. However, estimated reward signals often become uncertain and fail to train a reliable statistical model since the existing methods tend to solve hard optimization problems directly. Inspired by a first-order optimization method called mirror descent, this paper proposes to predict a sequence of reward functions, which are iterative solutions for a constrained convex problem. IRL solutions derived by mirror descent are tolerant to the uncertainty incurred by target density estimation since the amount of reward learning is regulated with respect to local geometric constraints. We prove that the proposed mirror descent update rule ensures robust minimization of a Bregman divergence in terms of a rigorous regret bound of O(1/T) for step sizes ηtt=1T. Our IRL method was applied on top of an adversarial framework, and it outperformed existing adversarial methods in an extensive suite of benchmarks.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 23 of its 65 references that are in the map. First posted on arXiv 20 Oct 2022.

Builds on

23 of 65 references.

Closest ancestors · 3

Same direction, newest first

  1. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matching9 Nov
  2. 2020Regularized Inverse Reinforcement Learning7 Oct
  3. 2018Maximum Causal Tsallis Entropy Imitation Learning22 May

Other references · 20

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2021Convex optimizationcited by 2,479
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  5. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 660
  6. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  7. 2019A Theory of Regularized Markov Decision Processescited by 279
  8. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
  9. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
  10. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 149
  11. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 124
  12. 2020Primal Wasserstein Imitation Learningcited by 120
8 more
  1. 2020Mirror Descent Policy Optimizationcited by 102
  2. 2017Robust Imitation of Diverse Behaviorscited by 85
  3. 2021Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergencecited by 83
  4. 2017Triple Generative Adversarial Netscited by 59
  5. 2017Dual Discriminator Generative Adversarial Netscited by 55
  6. 2013The Information Geometry of Mirror Descentcited by 53
  7. 2018Convergence of Online Mirror Descentcited by 3
  8. 2019BreGMN: scaled-Bregman Generative Modeling Networkscited by 2

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 3.

5 papers.

  1. 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Dualitycited by 4
  2. 2024Expert Proximity as Surrogate Rewards for Single Demonstration Imitation Learningcited by 2
  3. 2025Variational Online Mirror Descent for Robust Learning in Schr"odinger Bridgecited by 2
  4. 2025Mirror Descent Using the Tempesta Generalized Multi-parametric Logarithmscited by 1
  5. 2024Multi Task Inverse Reinforcement Learning for Common Sense Rewardcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Adversarial Imitation Learning On Aggregated Data
  2. 2024Rethinking Adversarial Inverse Reinforcement Learning: Policy Imitation, Transferable Reward Recovery and Algebraic Equilibrium Proof
  3. 2022Option-Aware Adversarial Inverse Reinforcement Learning for Robotic Control
  4. 2020Reparameterized Variational Divergence Minimization for Stable Imitation
  5. 2023Provably Efficient Adversarial Imitation Learning with Unknown Transitions
  6. 2021Meta-Adversarial Inverse Reinforcement Learning for Decision-making Tasks
  7. 2023Inverse Reinforcement Learning with the Average Reward Criterion
  8. 2019Wasserstein Adversarial Imitation Learning