# Efficient Sampling-Based Maximum Entropy Inverse Reinforcement Learning with Application to Autonomous Driving

arXiv: https://arxiv.org/abs/2006.13704 · HTML: https://mlsanity.com/paper/2006.13704/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in Imitation and offline RL for driving, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 6 of its 27 references that are in the map.
- First posted on arXiv 22 Jun 2020.

## Placed in

- Offline RL and imitation learning from data / [Imitation and offline RL for driving](https://mlsanity.com/offlinerl/imitation-and-offline-rl-for-driving/)

## Builds on (most cited first, 6 of 6)

- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps](https://mlsanity.com/paper/1910.03088/) (2019)
- [Online Planning Algorithms for POMDPs](https://mlsanity.com/paper/1401.3436/) (2014)
- [Probabilistic Prediction of Interactive Driving Behavior via Hierarchical Inverse Reinforcement Learning](https://mlsanity.com/paper/1809.02926/) (2018)
- [LESS is More: Rethinking Probabilistic Models of Human Behavior](https://mlsanity.com/paper/2001.04465/) (2020)
- [Courteous Autonomous Cars](https://mlsanity.com/paper/1808.02633/) (2018)

## Built on by (most cited first, 30 of 34)

- [Driving Behavior Modeling using Naturalistic Human Driving Data with Inverse Reinforcement Learning](https://mlsanity.com/paper/2010.03118/) (2020)
- [DTPP: Differentiable Joint Conditional Prediction and Cost Evaluation for Tree Policy Planning in Autonomous Driving](https://mlsanity.com/paper/2310.05885/) (2023)
- [When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning](https://mlsanity.com/paper/2302.07457/) (2023)
- [Machine Learning for Autonomous Vehicle's Trajectory Prediction: A comprehensive survey, Challenges, and Future Research Directions](https://mlsanity.com/paper/2307.07527/) (2023)
- [Towards Theoretical Understanding of Inverse Reinforcement Learning](https://mlsanity.com/paper/2304.12966/) (2023)
- [A Review of Reward Functions for Reinforcement Learning in the context of Autonomous Driving](https://mlsanity.com/paper/2405.01440/) (2024)
- [OPIRL: Sample Efficient Off-Policy Inverse Reinforcement Learning via Distribution Matching](https://mlsanity.com/paper/2109.04307/) (2021)
- [Regularized Inverse Reinforcement Learning](https://mlsanity.com/paper/2010.03691/) (2020)
- [Offline Imitation Learning with Variational Counterfactual Reasoning](https://mlsanity.com/paper/2310.04706/) (2023)
- [Learning Human Rewards by Inferring Their Latent Intelligence Levels in Multi-Agent Games: A Theory-of-Mind Approach with Application to Driving Data](https://mlsanity.com/paper/2103.04289/) (2021)
- [Blending Data-Driven Priors in Dynamic Games](https://mlsanity.com/paper/2402.14174/) (2024)
- [Foresight in Motion: Reinforcing Trajectory Prediction with Reward Heuristics](https://mlsanity.com/paper/2507.12083/) (2025)
- [Inverse Reinforcement Learning Based Stochastic Driver Behavior Learning](https://mlsanity.com/paper/2107.06344/) (2021)
- [Learning Reward Models for Cooperative Trajectory Planning with Inverse Reinforcement Learning and Monte Carlo Tree Search](https://mlsanity.com/paper/2202.06443/) (2022)
- [PP-TIL: Personalized Planning for Autonomous Driving with Instance-based Transfer Imitation Learning](https://mlsanity.com/paper/2407.18569/) (2024)
- [Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Review](https://mlsanity.com/paper/2411.10268/) (2024)
- [How To Not Drive: Learning Driving Constraints from Demonstration](https://mlsanity.com/paper/2110.00645/) (2021)
- [A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspective](https://mlsanity.com/paper/2503.23650/) (2025)
- [Calibration of Human Driving Behavior and Preference Using Naturalistic Traffic Data](https://mlsanity.com/paper/2105.01820/) (2021)
- [Active Inverse Learning in Stackelberg Trajectory Games](https://mlsanity.com/paper/2308.08017/) (2023)
- [Analyzing Closed-loop Training Techniques for Realistic Traffic Agent Models in Autonomous Highway Driving Simulations](https://mlsanity.com/paper/2410.15987/) (2024)
- [Offline Reinforcement Learning using Human-Aligned Reward Labeling for Autonomous Emergency Braking in Occluded Pedestrian Crossing](https://mlsanity.com/paper/2504.08704/) (2025)
- [Trust Region Reward Optimization and Proximal Inverse Reward Optimization Algorithm](https://mlsanity.com/paper/2509.23135/) (2025)
- [A Review of Learning-Based Motion Planning: Toward a Data-Driven Optimal Control Approach](https://mlsanity.com/paper/2512.11944/) (2025)
- [Human-Like Implicit Intention Expression for Autonomous Driving Motion Planning: A Method Based on Learning Human Intention Priors](https://mlsanity.com/paper/2307.15950/) (2023)
- [Identifying Reaction-Aware Driving Styles of Stochastic Model Predictive Controlled Vehicles by Inverse Reinforcement Learning](https://mlsanity.com/paper/2308.12069/) (2023)
- [ReMAV: Reward Modeling of Autonomous Vehicles for Finding Likely Failure Events](https://mlsanity.com/paper/2308.14550/) (2023)
- [Reinforced Imitative Trajectory Planning for Urban Automated Driving](https://mlsanity.com/paper/2410.15607/) (2024)
- [A physics-informed Bayesian optimization method for rapid development of electrical machines](https://mlsanity.com/paper/2503.00420/) (2025)
- [Recursive Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/2504.13241/) (2025)

## Abstract (arXiv)

In the past decades, we have witnessed significant progress in the domain of autonomous driving. Advanced techniques based on optimization and reinforcement learning (RL) become increasingly powerful at solving the forward problem: given designed reward/cost functions, how should we optimize them and obtain driving policies that interact with the environment safely and efficiently. Such progress has raised another equally important question: what should we optimize? Instead of manually specifying the reward functions, it is desired that we can extract what human drivers try to optimize from real traffic data and assign that to autonomous vehicles to enable more naturalistic and transparent interaction between humans and intelligent agents. To address this issue, we present an efficient sampling-based maximum-entropy inverse reinforcement learning (IRL) algorithm in this paper. Different from existing IRL algorithms, by introducing an efficient continuous-domain trajectory sampler, the proposed algorithm can directly learn the reward functions in the continuous domain while considering the uncertainties in demonstrated trajectories from human drivers. We evaluate the proposed algorithm on real driving data, including both non-interactive and interactive scenarios. The experimental results show that the proposed algorithm achieves more accurate prediction performance with faster convergence speed and better generalization compared to other baseline IRL algorithms.