# Inverse Reinforcement Learning with Just Classification and a Few Regressions

arXiv: https://arxiv.org/abs/2509.21172 · HTML: https://mlsanity.com/paper/2509.21172/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 29 of its 57 references that are in the map.
- First posted on arXiv 25 Sep 2025.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 29 of 29)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [Information-Theoretic Considerations in Batch Reinforcement Learning](https://mlsanity.com/paper/1905.00360/) (2019)
- [Bridging the Gap Between Value and Policy Based Reinforcement Learning](https://mlsanity.com/paper/1702.08892/) (2017)
- [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015)
- [Orthogonal Statistical Learning](https://mlsanity.com/paper/1901.09036/) (2019)
- [Minimax Weight and Q-Function Learning for Off-Policy Evaluation](https://mlsanity.com/paper/1910.12809/) (2019)
- [Imitation Learning as f-Divergence Minimization](https://mlsanity.com/paper/1905.12888/) (2019)
- [Offline Reinforcement Learning with Realizability and Single-policy Concentrability](https://mlsanity.com/paper/2202.04634/) (2022)
- [Batch Value-function Approximation with Only Realizability](https://mlsanity.com/paper/2008.04990/) (2020)
- [The Role of Coverage in Online Reinforcement Learning](https://mlsanity.com/paper/2210.04157/) (2022)
- [Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning](https://mlsanity.com/paper/2407.15007/) (2024)
- [Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation](https://mlsanity.com/paper/2111.10919/) (2021)
- [Removing Hidden Confounding by Experimental Grounding](https://mlsanity.com/paper/1810.11646/) (2018)
- [Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency](https://mlsanity.com/paper/2102.02981/) (2021)
- [Identifiability in inverse reinforcement learning](https://mlsanity.com/paper/2106.03498/) (2021)
- [Invariance in Policy Optimisation and Partial Identifiability in Reward Learning](https://mlsanity.com/paper/2203.07475/) (2022)
- [Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees](https://mlsanity.com/paper/2210.01808/) (2022)
- [A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learning](https://mlsanity.com/paper/2104.13844/) (2021)
- [Revisiting Maximum Entropy Inverse Reinforcement Learning: New Perspectives and Algorithms](https://mlsanity.com/paper/2012.00889/) (2020)
- [Offline Minimax Soft-Q-learning Under Realizability and Partial Coverage](https://mlsanity.com/paper/2302.02392/) (2023)
- [Revealed Preference at Scale: Learning Personalized Preferences from Assortment Choices](https://mlsanity.com/paper/1509.05113/) (2015)
- [Partial Identifiability and Misspecification in Inverse Reinforcement Learning](https://mlsanity.com/paper/2411.15951/) (2024)

## Built on by (most cited first, 5 of 5)

- [Bellman Calibration for V-Learning in Offline Reinforcement Learning](https://mlsanity.com/paper/2512.23694/) (2025)
- [Efficient Inference for Inverse Reinforcement Learning and Dynamic Discrete Choice Models](https://mlsanity.com/paper/2512.24407/) (2025)
- [Interactionless Inverse Reinforcement Learning: A Data-Centric Framework for Durable Alignment](https://mlsanity.com/paper/2602.14844/) (2026)
- [Fitted Occupancy-Ratio Evaluation without Bellman Completeness](https://mlsanity.com/paper/2607.05375/) (2026)
- [Scalable Multi-Task Inverse Reinforcement Learning](https://mlsanity.com/paper/2610.00758/) (2026)

## Abstract (arXiv)

Inverse reinforcement learning (IRL) aims to infer rewards from observed behavior, but rewards are not identified from the policy alone: many reward--value pairs can rationalize the same actions. Meaningful reward recovery therefore requires a normalization, yet existing normalized IRL methods often rely on anchor-action restrictions or specialized neural architectures. We study reward recovery in the maximum-entropy, or Gumbel-shock, model under a broad class of statewise affine normalizations, with anchor-action constraints as a special case. This yields Generalized Policy-to-Q-to-Reward (GenPQR), a modular procedure that estimates the behavior policy, evaluates its soft Q-function through the Bellman equation, and recovers the normalized reward. Both stages can be implemented with off-the-shelf classification and regression methods. We prove modular finite-sample guarantees under general function approximation, with separate policy-estimation and Q-estimation errors. As a concrete instantiation, we study GenPQR with fitted Q-evaluation, reducing IRL to policy estimation followed by regression. Experiments show that GenPQR matches or improves reward recovery relative to DeepPQR while remaining simpler and more modular. Compared with DeepPQR, our theory goes beyond anchor actions, accommodates large and continuous action spaces, makes coverage requirements explicit, and is not tied to a specific neural-network architecture or training procedure.