# Simulating Emergent Properties of Human Driving Behavior Using Multi-Agent Reward Augmented Imitation Learning

arXiv: https://arxiv.org/abs/1903.05766 · HTML: https://mlsanity.com/paper/1903.05766/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Imitation and offline RL for driving, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 6 of its 22 references that are in the map.
- First posted on arXiv 14 Mar 2019.

## Placed in

- Offline RL and imitation learning from data / [Imitation and offline RL for driving](https://mlsanity.com/offlinerl/imitation-and-offline-rl-for-driving/)

## Builds on (most cited first, 6 of 6)

- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Multi-Agent Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1807.09936/) (2018)
- [Imitating Driver Behavior with Generative Adversarial Networks](https://mlsanity.com/paper/1701.06699/) (2017)
- [Multi-Agent Imitation Learning for Driving Simulation](https://mlsanity.com/paper/1803.01044/) (2018)
- [RAIL: Risk-Averse Imitation Learning](https://mlsanity.com/paper/1707.06658/) (2017)

## Built on by (most cited first, 24 of 24)

- [TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors](https://mlsanity.com/paper/2101.06557/) (2021)
- [A Survey of Deep RL and IL for Autonomous Driving Policy Learning](https://mlsanity.com/paper/2101.01993/) (2021)
- [Modeling Human Driving Behavior through Generative Adversarial Imitation Learning](https://mlsanity.com/paper/2006.06412/) (2020)
- [TrajGen: Generating Realistic and Diverse Trajectories with Reactive and Feasible Agent Behaviors for Autonomous Driving](https://mlsanity.com/paper/2203.16792/) (2022)
- [Congestion-aware Multi-agent Trajectory Prediction for Collision Avoidance](https://mlsanity.com/paper/2103.14231/) (2021)
- [Residual Q-Learning: Offline and Online Policy Customization without Value](https://mlsanity.com/paper/2306.09526/) (2023)
- [LASIL: Learner-Aware Supervised Imitation Learning For Long-term Microscopic Traffic Simulation](https://mlsanity.com/paper/2403.17601/) (2024)
- [Diverse Controllable Diffusion Policy with Signal Temporal Logic](https://mlsanity.com/paper/2503.02924/) (2025)
- [Reactive and Safe Road User Simulations using Neural Barrier Certificates](https://mlsanity.com/paper/2109.06689/) (2021)
- [RITA: Boost Driving Simulators with Realistic Interactive Traffic Flow](https://mlsanity.com/paper/2211.03408/) (2022)
- [Grounded Relational Inference: Domain Knowledge Driven Explainable Autonomous Driving](https://mlsanity.com/paper/2102.11905/) (2021)
- [SHAIL: Safety-Aware Hierarchical Adversarial Imitation Learning for Autonomous Driving in Urban Environments](https://mlsanity.com/paper/2204.01922/) (2022)
- [DecompGAIL: Learning Realistic Traffic Behaviors with Decomposed Multi-Agent Generative Adversarial Imitation Learning](https://mlsanity.com/paper/2510.06913/) (2025)
- [Exploring the trade off between human driving imitation and safety for traffic simulation](https://mlsanity.com/paper/2208.04803/) (2022)
- [ConBaT: Control Barrier Transformer for Safe Policy Learning](https://mlsanity.com/paper/2303.04212/) (2023)
- [Long-term Microscopic Traffic Simulation with History-Masked Multi-agent Imitation Learning](https://mlsanity.com/paper/2306.06401/) (2023)
- [Fail-Safe Adversarial Generative Imitation Learning](https://mlsanity.com/paper/2203.01696/) (2022)
- [Semi-Supervised Imitation Learning of Team Policies from Suboptimal Demonstrations](https://mlsanity.com/paper/2205.02959/) (2022)
- [Online Context Learning for Socially Compliant Navigation](https://mlsanity.com/paper/2406.11495/) (2024)
- [Hierarchical Imitation Learning of Team Behavior from Heterogeneous Demonstrations](https://mlsanity.com/paper/2502.17618/) (2025)
- [INTAGS: Interactive Agent-Guided Simulation](https://mlsanity.com/paper/2309.01784/) (2023)
- [LitSim: A Conflict-aware Policy for Long-term Interactive Traffic Simulation](https://mlsanity.com/paper/2403.04299/) (2024)
- [Analyzing Closed-loop Training Techniques for Realistic Traffic Agent Models in Autonomous Highway Driving Simulations](https://mlsanity.com/paper/2410.15987/) (2024)
- [Artificial Intelligence for Modeling and Simulation of Mixed Automated and Human Traffic](https://mlsanity.com/paper/2604.12857/) (2026)

## Abstract (arXiv)

Recent developments in multi-agent imitation learning have shown promising results for modeling the behavior of human drivers. However, it is challenging to capture emergent traffic behaviors that are observed in real-world datasets. Such behaviors arise due to the many local interactions between agents that are not commonly accounted for in imitation learning. This paper proposes Reward Augmented Imitation Learning (RAIL), which integrates reward augmentation into the multi-agent imitation learning framework and allows the designer to specify prior knowledge in a principled fashion. We prove that convergence guarantees for the imitation learning process are preserved under the application of reward augmentation. This method is validated in a driving scenario, where an entire traffic scene is controlled by driving policies learned using our proposed algorithm. Further, we demonstrate improved performance in comparison to traditional imitation learning algorithms both in terms of the local actions of a single agent and the behavior of emergent properties in complex, multi-agent settings.