# Driving Behavior Modeling using Naturalistic Human Driving Data with Inverse Reinforcement Learning

arXiv: https://arxiv.org/abs/2010.03118 · HTML: https://mlsanity.com/paper/2010.03118/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Placed in Imitation and offline RL for driving, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 9 of its 34 references that are in the map.
- First posted on arXiv 7 Oct 2020.

## Placed in

- Offline RL and imitation learning from data / [Imitation and offline RL for driving](https://mlsanity.com/offlinerl/imitation-and-offline-rl-for-driving/)

## Builds on (most cited first, 9 of 9)

- [Multi-modal Sensor Fusion-Based Deep Neural Network for End-to-end Autonomous Driving with Scene Understanding](https://mlsanity.com/paper/2005.09202/) (2020)
- [Efficient Sampling-Based Maximum Entropy Inverse Reinforcement Learning with Application to Autonomous Driving](https://mlsanity.com/paper/2006.13704/) (2020)
- [Driving with Style: Inverse Reinforcement Learning in General-Purpose Planning for Automated Driving](https://mlsanity.com/paper/1905.00229/) (2019)
- [Probabilistic Prediction of Interactive Driving Behavior via Hierarchical Inverse Reinforcement Learning](https://mlsanity.com/paper/1809.02926/) (2018)
- [Interaction-Aware Trajectory Prediction of Connected Vehicles using CNN-LSTM Networks](https://mlsanity.com/paper/2005.12134/) (2020)
- [Learning from Naturalistic Driving Data for Human-like Autonomous Highway Driving](https://mlsanity.com/paper/2005.11470/) (2020)
- [Generic Prediction Architecture Considering both Rational and Irrational Driving Behaviors](https://mlsanity.com/paper/1907.10170/) (2019)
- [Interpretable Modelling of Driving Behaviors in Interactive Driving Scenarios based on Cumulative Prospect Theory](https://mlsanity.com/paper/1907.08707/) (2019)
- [Expressing Diverse Human Driving Behavior with Probabilistic Rewards and Online Inference](https://mlsanity.com/paper/2008.08812/) (2020)

## Built on by (most cited first, 30 of 46)

- [Differentiable Integrated Motion Prediction and Planning with Learnable Cost Function for Autonomous Driving](https://mlsanity.com/paper/2207.10422/) (2022)
- [Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Driving](https://mlsanity.com/paper/2103.10690/) (2021)
- [DTPP: Differentiable Joint Conditional Prediction and Cost Evaluation for Tree Policy Planning in Autonomous Driving](https://mlsanity.com/paper/2310.05885/) (2023)
- [CitySim: A Drone-Based Vehicle Trajectory Dataset for Safety Oriented Research and Digital Twins](https://mlsanity.com/paper/2208.11036/) (2022)
- [Gen-Drive: Enhancing Diffusion Generative Driving Policies with Reward Modeling and Reinforcement Learning Fine-tuning](https://mlsanity.com/paper/2410.05582/) (2024)
- [Conditional Predictive Behavior Planning with Inverse Reinforcement Learning for Human-like Autonomous Driving](https://mlsanity.com/paper/2212.08787/) (2022)
- [Improved Deep Reinforcement Learning with Expert Demonstrations for Urban Autonomous Driving](https://mlsanity.com/paper/2102.09243/) (2021)
- [Augmenting Reinforcement Learning with Transformer-based Scene Representation Learning for Decision-making of Autonomous Driving](https://mlsanity.com/paper/2208.12263/) (2022)
- [TrajGen: Generating Realistic and Diverse Trajectories with Reactive and Feasible Agent Behaviors for Autonomous Driving](https://mlsanity.com/paper/2203.16792/) (2022)
- [Driving in Real Life with Inverse Reinforcement Learning](https://mlsanity.com/paper/2206.03004/) (2022)
- [Adversarial Safety-Critical Scenario Generation using Naturalistic Human Driving Priors](https://mlsanity.com/paper/2408.03200/) (2024)
- [DistillDrive: End-to-End Multi-Mode Autonomous Driving Distillation by Isomorphic Hetero-Source Planning Model](https://mlsanity.com/paper/2508.05402/) (2025)
- [Driver Digital Twin for Online Prediction of Personalized Lane Change Behavior](https://mlsanity.com/paper/2211.01294/) (2022)
- [STrajNet: Multi-modal Hierarchical Transformer for Occupancy Flow Field Prediction in Autonomous Driving](https://mlsanity.com/paper/2208.00394/) (2022)
- [Occupancy Prediction-Guided Neural Planner for Autonomous Driving](https://mlsanity.com/paper/2305.03303/) (2023)
- [Cooperative Driving of Connected Autonomous Vehicles in Heterogeneous Mixed Traffic: A Game Theoretic Approach](https://mlsanity.com/paper/2305.03563/) (2023)
- [Generating and Evolving Reward Functions for Highway Driving with Large Language Models](https://mlsanity.com/paper/2406.10540/) (2024)
- [Learning Interaction-aware Motion Prediction Model for Decision-making in Autonomous Driving](https://mlsanity.com/paper/2302.03939/) (2023)
- [Hypergraph-based Motion Generation with Multi-modal Interaction Relational Reasoning](https://mlsanity.com/paper/2409.11676/) (2024)
- [PDB: Not All Drivers Are the Same -- A Personalized Dataset for Understanding Driving Behavior](https://mlsanity.com/paper/2503.06477/) (2025)
- [From Words to Wheels: Automated Style-Customized Policy Generation for Autonomous Driving](https://mlsanity.com/paper/2409.11694/) (2024)
- [Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experience](https://mlsanity.com/paper/2208.04919/) (2022)
- [Importance Sampling-Guided Meta-Training for Intelligent Agents in Highly Interactive Environments](https://mlsanity.com/paper/2407.15839/) (2024)
- [PP-TIL: Personalized Planning for Autonomous Driving with Instance-based Transfer Imitation Learning](https://mlsanity.com/paper/2407.18569/) (2024)
- [HAD-Gen: Human-like and Diverse Driving Behavior Modeling for Controllable Scenario Generation](https://mlsanity.com/paper/2503.15049/) (2025)
- [Multi-Objective Reinforcement Learning for Adaptable Personalized Autonomous Driving](https://mlsanity.com/paper/2505.05223/) (2025)
- [An Adaptive Human Driver Model for Realistic Race Car Simulations](https://mlsanity.com/paper/2203.01909/) (2022)
- [Interaction-Aware Personalized Vehicle Trajectory Prediction Using Temporal Graph Neural Networks](https://mlsanity.com/paper/2308.07439/) (2023)
- [Accelerating the Evolution of Personalized Automated Lane Change through Lesson Learning](https://mlsanity.com/paper/2405.07543/) (2024)
- [When Context Is Not Enough: Modeling Unexplained Variability in Car-Following Behavior](https://mlsanity.com/paper/2507.07012/) (2025)

## Abstract (arXiv)

Driving behavior modeling is of great importance for designing safe, smart, and personalized autonomous driving systems. In this paper, an internal reward function-based driving model that emulates the human's decision-making mechanism is utilized. To infer the reward function parameters from naturalistic human driving data, we propose a structural assumption about human driving behavior that focuses on discrete latent driving intentions. It converts the continuous behavior modeling problem to a discrete setting and thus makes maximum entropy inverse reinforcement learning (IRL) tractable to learn reward functions. Specifically, a polynomial trajectory sampler is adopted to generate candidate trajectories considering high-level intentions and approximate the partition function in the maximum entropy IRL framework. An environment model considering interactive behaviors among the ego and surrounding vehicles is built to better estimate the generated trajectories. The proposed method is applied to learn personalized reward functions for individual human drivers from the NGSIM highway driving dataset. The qualitative results demonstrate that the learned reward functions are able to explicitly express the preferences of different drivers and interpret their decisions. The quantitative results reveal that the learned reward functions are robust, which is manifested by only a marginal decline in proximity to the human driving trajectories when applying the reward function in the testing conditions. For the testing performance, the personalized modeling method outperforms the general modeling approach, significantly reducing the modeling errors in human likeness (a custom metric to gauge accuracy), and these two methods deliver better results compared to other baseline methods.