mlsanityData to 7 Oct 2026

Convergence of a model-free entropy-regularized inverse reinforcement learning algorithm

arXiv 2403.16829 ↗First posted 25 Mar 2024cs.LG cs.AIcited by 2

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Given a dataset of expert demonstrations, inverse reinforcement learning (IRL) aims to recover a reward for which the expert is optimal. This work proposes a model-free algorithm to solve entropy-regularized IRL problem. In particular, we employ a stochastic gradient descent update for the reward and a stochastic soft policy iteration update for the policy. Assuming access to a generative model, we prove that our algorithm is guaranteed to recover a reward for which the expert is varepsilon-optimal using O(1/varepsilon2) samples of the Markov decision process (MDP). Furthermore, with O(1/varepsilon4) samples we prove that the optimal policy corresponding to the recovered reward is varepsilon-close to the expert policy in total variation distance.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 13 of its 27 references that are in the map. First posted on arXiv 25 Mar 2024.

Builds on

13 of 27 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Identifiability and Generalizability in Constrained Inverse Reinforcement Learning1 Jun
  2. 2022Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees4 Oct

Other references · 11

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2016Generative Adversarial Imitation Learningcited by 1,576
  3. 2019A Theory of Regularized Markov Decision Processescited by 279
  4. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 258
  5. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 170
  6. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 149
  7. 2020Logistic Q-Learningcited by 34
  8. 2021A Dual Approach to Constrained Markov Decision Processes with Entropy Regularizationcited by 33
  9. 2021Online Apprenticeship Learningcited by 28
  10. 2022Proximal Point Imitation Learningcited by 25
  11. 2015Information-Theoretic Bounded Rationalitycited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 1.

2 papers.

  1. 2024Maximum Entropy Inverse Reinforcement Learning of Diffusion Models with Energy-Based Modelscited by 4
  2. 2026Interactive Inverse Reinforcement Learning of Interaction Scenarios via Bi-level Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Statistical analysis of Inverse Entropy-regularized Reinforcement Learning
  2. 2020Regularized Inverse Reinforcement Learning
  3. 2020Revisiting Maximum Entropy Inverse Reinforcement Learning: New Perspectives and Algorithms
  4. 2022Weighted Maximum Entropy Inverse Reinforcement Learning
  5. 2023Inverse Reinforcement Learning with the Average Reward Criterion
  6. 2026Fast Rates for Inverse Reinforcement Learning
  7. 2019Generalized Maximum Causal Entropy for Inverse Reinforcement Learning
  8. 2021Inverse Reinforcement Learning with Explicit Policy Estimates