mlsanityData to 7 Oct 2026

Inverse Reinforcement Learning with the Average Reward Criterion

arXiv 2305.14608 ↗First posted 24 May 2023cs.LG cs.AIcited by 8

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

We study the problem of Inverse Reinforcement Learning (IRL) with an average-reward criterion. The goal is to recover an unknown policy and a reward function when the agent only has samples of states and actions from an experienced agent. Previous IRL methods assume that the expert is trained in a discounted environment, and the discount factor is known. This work alleviates this assumption by proposing an average-reward framework with efficient learning algorithms. We develop novel stochastic first-order methods to solve the IRL problem under the average-reward setting, which requires solving an Average-reward Markov Decision Process (AMDP) as a subproblem. To solve the subproblem, we develop a Stochastic Policy Mirror Descent (SPMD) method under general state and action spaces that needs mathcalO(1/varepsilon) steps of gradient computation. Equipped with SPMD, we propose the Inverse Policy Mirror Descent (IPMD) method for solving the IRL problem with a O(1/varepsilon2) complexity. To the best of our knowledge, the aforementioned complexity results are new in IRL. Finally, we corroborate our analysis with numerical experiments using the MuJoCo benchmark and additional control tasks.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 22 of its 41 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

22 of 41 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Towards Theoretical Understanding of Inverse Reinforcement Learning25 Apr
  2. 2022Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees4 Oct
  3. 2021Scalable Bayesian Inverse Reinforcement Learning12 Feb
  4. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matching9 Nov

Other references · 18

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016OpenAI Gymcited by 1,684
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  5. 2017One-Shot Imitation Learningcited by 232
  6. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 191
  7. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 170
  8. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 149
  9. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 108
  10. 2017Robust Imitation of Diverse Behaviorscited by 85
  11. 2021Reward is enough for convex MDPscited by 82
  12. 2021Concave Utility Reinforcement Learning: the Mean-Field Game Viewpointcited by 54
6 more
  1. 2019Task-Relevant Adversarial Imitation Learningcited by 45
  2. 2022Stochastic first-order methods for average-reward Markov decision processescited by 30
  3. 2021Finite-Sample Analysis of Off-Policy Natural Actor-Critic Algorithmcited by 26
  4. 2021Finite-Sample Analysis of Off-Policy Natural Actor-Critic with Linear Function Approximationcited by 22
  5. 2022Policy Optimization over General State and Action Spacescited by 21
  6. 2019Unknown mixing times in apprenticeship and reinforcement learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 3; 2026: 1.

8 papers.

  1. 2024Imitation Learning in Discounted Linear MDPs without exploration assumptionscited by 15
  2. 2025Average-Reward Soft Actor-Criticcited by 6
  3. 2025Inverse Reinforcement Learning with Switching Rewards and History Dependency for Characterizing Animal Behaviorscited by 6
  4. 2023Infer and Adapt: Bipedal Locomotion Reward Learning from Demonstrations via Inverse Reinforcement Learningcited by 5
  5. 2024Learn to Teach: Sample-Efficient Privileged Learning for Humanoid Locomotion over Diverse Terrainscited by 4
  6. 2024Enhancing Spectrum Efficiency in 6G Satellite Networks: A GAIL-Powered Policy Learning via Asynchronous Federated Inverse Reinforcement Learningcited by 2
  7. 2025Distributional Inverse Reinforcement Learningcited by 0
  8. 2026Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games with Average Rewardcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Convergence of a model-free entropy-regularized inverse reinforcement learning algorithm
  2. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  3. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
  4. 2025Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroids
  5. 2025Apprenticeship learning with prior beliefs using inverse optimization
  6. 2019On the Correctness and Sample Complexity of Inverse Reinforcement Learning
  7. 2025Inverse Reinforcement Learning via Convex Optimization
  8. 2020Regularized Inverse Reinforcement Learning