# Truly Batch Apprenticeship Learning with Deep Successor Features

arXiv: https://arxiv.org/abs/1903.10077 · HTML: https://mlsanity.com/paper/1903.10077/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Mar 2019.

## Built on by (most cited first, 27 of 27)

- [IQ-Learn: Inverse soft-Q Learning for Imitation](https://mlsanity.com/paper/2106.12142/) (2021)
- [Scalable Bayesian Inverse Reinforcement Learning](https://mlsanity.com/paper/2102.06483/) (2021)
- [Benchmarks and Algorithms for Offline Preference-Based Reward Learning](https://mlsanity.com/paper/2301.01392/) (2023)
- [Strictly Batch Imitation Learning by Energy-based Distribution Matching](https://mlsanity.com/paper/2006.14154/) (2020)
- [When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning](https://mlsanity.com/paper/2302.07457/) (2023)
- [PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning](https://mlsanity.com/paper/2102.12560/) (2021)
- [CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learning](https://mlsanity.com/paper/2302.04782/) (2023)
- [Benchmarking Constraint Inference in Inverse Reinforcement Learning](https://mlsanity.com/paper/2206.09670/) (2022)
- [Offline Preference-Based Apprenticeship Learning](https://mlsanity.com/paper/2107.09251/) (2021)
- [Dynamics-Aware Comparison of Learned Reward Functions](https://mlsanity.com/paper/2201.10081/) (2022)
- [Inverse Reinforcement Learning in Contextual MDPs](https://mlsanity.com/paper/1905.09710/) (2019)
- [Inverse Contextual Bandits: Learning How Behavior Evolves over Time](https://mlsanity.com/paper/2107.06317/) (2021)
- [How to Leverage Diverse Demonstrations in Offline Imitation Learning](https://mlsanity.com/paper/2405.17476/) (2024)
- [Inverse Decision Modeling: Learning Interpretable Representations of Behavior](https://mlsanity.com/paper/2310.18591/) (2023)
- [Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching](https://mlsanity.com/paper/2411.07007/) (2024)
- [Successor Feature Sets: Generalizing Successor Representations Across Policies](https://mlsanity.com/paper/2103.02650/) (2021)
- [Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experience](https://mlsanity.com/paper/2208.04919/) (2022)
- [Predictive representations: building blocks of intelligence](https://mlsanity.com/paper/2402.06590/) (2024)
- [Markov Balance Satisfaction Improves Performance in Strictly Batch Offline Imitation Learning](https://mlsanity.com/paper/2408.09125/) (2024)
- [Conditional Kernel Imitation Learning for Continuous State Environments](https://mlsanity.com/paper/2308.12573/) (2023)
- [Language Model Distillation: A Temporal Difference Imitation Learning Perspective](https://mlsanity.com/paper/2505.20335/) (2025)
- [A Generalized Apprenticeship Learning Framework for Capturing Evolving Student Pedagogical Strategies](https://mlsanity.com/paper/2602.20527/) (2026)
- [Replicating Complex Dialogue Policy of Humans via Offline Imitation Learning with Supervised Regularization](https://mlsanity.com/paper/2305.03987/) (2023)
- [OMG-RL:Offline Model-based Guided Reward Learning for Heparin Treatment](https://mlsanity.com/paper/2409.13299/) (2024)
- [Extracting Expert's Goals by What-if Interpretable Modeling](https://mlsanity.com/paper/2110.15165/) (2021)
- [A Context Engineering Framework for Improving Enterprise AI Agents based on Digital-Twin MDP](https://mlsanity.com/paper/2603.22083/) (2026)
- [Quantifying Potential Observation Missingness in Inverse Reinforcement Learning](https://mlsanity.com/paper/2605.12831/) (2026)

## Abstract (arXiv)

We introduce a novel apprenticeship learning algorithm to learn an expert's underlying reward structure in off-policy model-free batch settings. Unlike existing methods that require a dynamics model or additional data acquisition for on-policy evaluation, our algorithm requires only the batch data of observed expert behavior. Such settings are common in real-world tasks---health care, finance or industrial processes ---where accurate simulators do not exist or data acquisition is costly. To address challenges in batch settings, we introduce Deep Successor Feature Networks(DSFN) that estimate feature expectations in an off-policy setting and a transition-regularized imitation network that produces a near-expert initial policy and an efficient feature representation. Our algorithm achieves superior results in batch settings on both control benchmarks and a vital clinical task of sepsis management in the Intensive Care Unit.