mlsanityData to 13 Oct 2026

Statistical analysis of Inverse Entropy-regularized Reinforcement Learning

arXiv 2512.06956 ↗First posted 7 Dec 2025stat.ML cs.LG math.STcited by 0

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

also in 1 more
Abstract (arXiv)

Inverse reinforcement learning aims to infer the reward function that explains expert behavior observed through trajectories of state--action pairs. A long-standing difficulty in classical IRL is the non-uniqueness of the recovered reward: many reward functions can induce the same optimal policy, rendering the inverse problem ill-posed. In this paper, we develop a statistical framework for Inverse Entropy-regularized Reinforcement Learning that resolves this ambiguity by combining entropy regularization with a least-squares reconstruction of the reward from the soft Bellman residual. This combination yields a unique and well-defined so-called least-squares reward consistent with the expert policy. We model the expert demonstrations as a Markov chain with the invariant distribution defined by an unknown expert policy π^star and estimate the policy by a penalized maximum-likelihood procedure over a class of conditional distributions on the action space. We establish high-probability bounds for the excess Kullback--Leibler divergence between the estimated policy and the expert policy, accounting for statistical complexity through covering numbers of the policy class. These results lead to non-asymptotic minimax optimal convergence rates for the least-squares reward function, revealing the interplay between smoothing (entropy regularization), model complexity, and sample size. Our analysis bridges the gap between behavior cloning, inverse reinforcement learning, and modern statistical learning theory.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 22 of its 45 references that are in the map. First posted on arXiv 7 Dec 2025.

Builds on

22 of 45 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learning3 Jun
  2. 2023Identifiability and Generalizability in Constrained Inverse Reinforcement Learning1 Jun
  3. 2023Towards Theoretical Understanding of Inverse Reinforcement Learning25 Apr
  4. 2021Identifiability in inverse reinforcement learning7 Jun
  5. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress18 Jun

Other references · 17

  1. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 660
  2. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 432
  3. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  4. 2019A Theory of Regularized Markov Decision Processescited by 281
  5. 2017A unified view of entropy-regularized Markov decision processescited by 177
  6. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 175
  7. 2018Understanding the impact of entropy on policy optimizationcited by 172
  8. 2020Finite Time Analysis of Linear Two-timescale Stochastic Approximation with Markovian Noisecited by 52
  9. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  10. 2023Foundations of Reinforcement Learning and Interactive Decision Makingcited by 33
  11. 2019Finite-Time Analysis and Restarting Scheme for Linear Two-Time-Scale Stochastic Approximationcited by 18
  12. 2023Tight Finite Time Bounds of Two-Time-Scale Linear Stochastic Approximation with Markovian Noisecited by 14
5 more
  1. 2023Demonstration-Regularized RLcited by 8
  2. 2023Rosenthal-type inequalities for linear statistics of Markov chainscited by 5
  3. 2024Two-Timescale Linear Stochastic Approximation: Constant Stepsizes Go a Long Waycited by 5
  4. 2025Gaussian Approximation for Two-Timescale Linear Stochastic Approximationcited by 5
  5. 2022Model-Based Imitation Learning Using Entropy Regularization of Model and Policycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Convergence of a model-free entropy-regularized inverse reinforcement learning algorithm
  2. 2020Revisiting Maximum Entropy Inverse Reinforcement Learning: New Perspectives and Algorithms
  3. 2020Regularized Inverse Reinforcement Learning
  4. 2022Weighted Maximum Entropy Inverse Reinforcement Learning
  5. 2019Generalized Maximum Causal Entropy for Inverse Reinforcement Learning
  6. 2026Fast Rates for Inverse Reinforcement Learning
  7. 2026Decoding Rewards in Competitive Games: Inverse Game Theory with Entropy Regularization
  8. 2025Inverse Reinforcement Learning with Just Classification and a Few Regressions