mlsanityData to 7 Oct 2026

Apprenticeship learning with prior beliefs using inverse optimization

arXiv 2505.21639 ↗First posted 27 May 2025cs.LG math.OCcited by 0

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

The relationship between inverse reinforcement learning (IRL) and inverse optimization (IO) for Markov decision processes (MDPs) has been relatively underexplored in the literature, despite addressing the same problem. In this work, we revisit the relationship between the IO framework for MDPs, IRL, and apprenticeship learning (AL). We incorporate prior beliefs on the structure of the cost function into the IRL and AL problems, and demonstrate that the convex-analytic view of the AL formalism emerges as a relaxation of our framework. Notably, the AL formalism is a special case in our framework when the regularization term is absent. Focusing on the suboptimal expert setting, we formulate the AL problem as a regularized min-max problem. The regularizer plays a key role in addressing the ill-posedness of IRL by guiding the search for plausible cost functions. To solve the resulting regularized-convex-concave-min-max problem, we use stochastic mirror descent (SMD) and establish convergence bounds for the proposed method. Numerical experiments highlight the critical role of regularization in learning cost vectors and apprentice policies.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 12 of its 27 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

12 of 27 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Reward Compatibility: A Framework for Inverse RL14 Jan
  2. 2023Towards Theoretical Understanding of Inverse Reinforcement Learning25 Apr
  3. 2022Active Exploration for Inverse Reinforcement Learning18 Jul
  4. 2021Efficient Performance Bounds for Primal-Dual Reinforcement Learning from Demonstrations28 Dec
  5. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations12 Apr

Other references · 7

  1. 2016Concrete Problems in AI Safetycited by 1,582
  2. 2016Generative Adversarial Imitation Learningcited by 1,576
  3. 2017Inverse Reward Designcited by 246
  4. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regressioncited by 84
  5. 2020Efficiently Solving MDPs with Stochastic Mirror Descentcited by 52
  6. 2021Inverse Optimization: Theory and Applicationscited by 48
  7. 2019Variance Reduction for Matrix Gamescited by 37

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Stochastic convex optimization for provably efficient apprenticeship learning
  2. 2025Inverse Reinforcement Learning via Convex Optimization
  3. 2023Inverse Reinforcement Learning with the Average Reward Criterion
  4. 2023A Novel Variational Lower Bound for Inverse Reinforcement Learning
  5. 2026Fast Rates for Inverse Reinforcement Learning
  6. 2024Convergence of a model-free entropy-regularized inverse reinforcement learning algorithm
  7. 2019Stochastic Inverse Reinforcement Learning
  8. 2021A Generalised Inverse Reinforcement Learning Framework