mlsanityData to 14 Oct 2026

Inverse Reinforcement Learning with Just Classification and a Few Regressions

arXiv 2509.21172 ↗First posted 25 Sep 2025cs.LG econ.EM math.OCcited by 5

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

also in 1 more
Abstract (arXiv)

Inverse reinforcement learning (IRL) aims to infer rewards from observed behavior, but rewards are not identified from the policy alone: many reward--value pairs can rationalize the same actions. Meaningful reward recovery therefore requires a normalization, yet existing normalized IRL methods often rely on anchor-action restrictions or specialized neural architectures. We study reward recovery in the maximum-entropy, or Gumbel-shock, model under a broad class of statewise affine normalizations, with anchor-action constraints as a special case. This yields Generalized Policy-to-Q-to-Reward (GenPQR), a modular procedure that estimates the behavior policy, evaluates its soft Q-function through the Bellman equation, and recovers the normalized reward. Both stages can be implemented with off-the-shelf classification and regression methods. We prove modular finite-sample guarantees under general function approximation, with separate policy-estimation and Q-estimation errors. As a concrete instantiation, we study GenPQR with fitted Q-evaluation, reducing IRL to policy estimation followed by regression. Experiments show that GenPQR matches or improves reward recovery relative to DeepPQR while remaining simpler and more modular. Compared with DeepPQR, our theory goes beyond anchor actions, accommodates large and continuous action spaces, makes coverage requirements explicit, and is not tied to a specific neural-network architecture or training procedure.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 57 references that are in the map. First posted on arXiv 25 Sep 2025.

Builds on

29 of 57 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Partial Identifiability and Misspecification in Inverse Reinforcement Learning24 Nov
  2. 2022Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees4 Oct
  3. 2021Identifiability in inverse reinforcement learning7 Jun
  4. 2020Revisiting Maximum Entropy Inverse Reinforcement Learning: New Perspectives and Algorithms1 Dec

Other references · 25

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2016Generative Adversarial Imitation Learningcited by 1,597
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  6. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  7. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  8. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  9. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  10. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 182
  11. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 176
  12. 2019Orthogonal Statistical Learningcited by 151
13 more
  1. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  2. 2019Imitation Learning as f-Divergence Minimizationcited by 123
  3. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 123
  4. 2020Batch Value-function Approximation with Only Realizabilitycited by 104
  5. 2022The Role of Coverage in Online Reinforcement Learningcited by 83
  6. 2024Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learningcited by 74
  7. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximationcited by 71
  8. 2018Removing Hidden Confounding by Experimental Groundingcited by 64
  9. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiencycited by 56
  10. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  11. 2021A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learningcited by 26
  12. 2023Offline Minimax Soft-Q-learning Under Realizability and Partial Coveragecited by 10
  13. 2015Revealed Preference at Scale: Learning Personalized Preferences from Assortment Choicescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Bellman Calibration for V-Learning in Offline Reinforcement Learningcited by 3
  2. 2025Efficient Inference for Inverse Reinforcement Learning and Dynamic Discrete Choice Modelscited by 3
  3. 2026Interactionless Inverse Reinforcement Learning: A Data-Centric Framework for Durable Alignmentcited by 1
  4. 2026Fitted Occupancy-Ratio Evaluation without Bellman Completenesscited by 1
  5. 2026Scalable Multi-Task Inverse Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Loop-Free Inverse Reinforcement Learning via Sequential Value Recovery with Q-Score Matching
  2. 2020Deep PQR: Solving Inverse Reinforcement Learning using Anchor Actions
  3. 2025Statistical analysis of Inverse Entropy-regularized Reinforcement Learning
  4. 2020Deep Inverse Q-learning with Constraints
  5. 2019Generalized Maximum Causal Entropy for Inverse Reinforcement Learning
  6. 2021Versatile Inverse Reinforcement Learning via Cumulative Rewards
  7. 2026Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approach
  8. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective