# Beyond RMSE: Do machine-learned models of road user interaction produce human-like behavior?

arXiv: https://arxiv.org/abs/2206.11110 · HTML: https://mlsanity.com/paper/2206.11110/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 56 references that are in the map.
- First posted on arXiv 22 Jun 2022.

## Builds on (most cited first, 17 of 17)

- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [CoverNet: Multimodal Behavior Prediction using Trajectory Sets](https://mlsanity.com/paper/1911.10298/) (2019)
- [Convolutional Social Pooling for Vehicle Trajectory Prediction](https://mlsanity.com/paper/1805.06771/) (2018)
- [Multiple Futures Prediction](https://mlsanity.com/paper/1911.00997/) (2019)
- [Scene Transformer: A unified architecture for predicting multiple agent trajectories](https://mlsanity.com/paper/2106.08417/) (2021)
- [TrafficPredict: Trajectory Prediction for Heterogeneous Traffic-Agents](https://mlsanity.com/paper/1811.02146/) (2018)
- [M2I: From Factored Marginal Trajectory Prediction to Interactive Prediction](https://mlsanity.com/paper/2202.11884/) (2022)
- [GRIP++: Enhanced Graph-based Interaction-aware Trajectory Prediction for Autonomous Driving](https://mlsanity.com/paper/1907.07792/) (2019)
- [Pedestrian Models for Autonomous Driving Part II: High-Level Models of Human Behavior](https://mlsanity.com/paper/2003.11959/) (2020)
- [Joint Interaction and Trajectory Prediction for Autonomous Driving using Graph Neural Networks](https://mlsanity.com/paper/1912.07882/) (2019)
- [DenseTNT: Waymo Open Dataset Motion Prediction Challenge 1st Place Solution](https://mlsanity.com/paper/2106.14160/) (2021)
- [Maneuver-based Trajectory Prediction for Self-driving Cars Using Spatio-temporal Convolutional Networks](https://mlsanity.com/paper/2109.07365/) (2021)
- [Pedestrian Models for Autonomous Driving Part I: Low-Level Models, from Sensing to Tracking](https://mlsanity.com/paper/2002.11669/) (2020)
- [Pedestrian Behavior Prediction for Automated Driving: Requirements, Metrics, and Relevant Features](https://mlsanity.com/paper/2012.08418/) (2020)
- [Flash: Fast and Light Motion Prediction for Autonomous Driving with Bayesian Inverse Planning and Learned Motion Profiles](https://mlsanity.com/paper/2203.08251/) (2022)
- [A human factors approach to validating driver models for interaction-aware automated vehicles](https://mlsanity.com/paper/2109.13077/) (2021)
- [Comparing merging behaviors observed in naturalistic data with behaviors generated by a machine learned model](https://mlsanity.com/paper/2104.10496/) (2021)

## Built on by (most cited first, 3 of 3)

- [Benchmark for Models Predicting Human Behavior in Gap Acceptance Scenarios](https://mlsanity.com/paper/2211.05455/) (2022)
- [Modeling human road crossing decisions as reward maximization with visual perception limitations](https://mlsanity.com/paper/2301.11737/) (2023)
- [Pedestrian crossing decisions can be explained by bounded optimal decision-making under noisy visual perception](https://mlsanity.com/paper/2402.04370/) (2024)

## Abstract (arXiv)

Autonomous vehicles use a variety of sensors and machine-learned models to predict the behavior of surrounding road users. Most of the machine-learned models in the literature focus on quantitative error metrics like the root mean square error (RMSE) to learn and report their models' capabilities. This focus on quantitative error metrics tends to ignore the more important behavioral aspect of the models, raising the question of whether these models really predict human-like behavior. Thus, we propose to analyze the output of machine-learned models much like we would analyze human data in conventional behavioral research. We introduce quantitative metrics to demonstrate presence of three different behavioral phenomena in a naturalistic highway driving dataset: 1) The kinematics-dependence of who passes a merging point first 2) Lane change by an on-highway vehicle to accommodate an on-ramp vehicle 3) Lane changes by vehicles on the highway to avoid lead vehicle conflicts. Then, we analyze the behavior of three machine-learned models using the same metrics. Even though the models' RMSE value differed, all the models captured the kinematic-dependent merging behavior but struggled at varying degrees to capture the more nuanced courtesy lane change and highway lane change behavior. Additionally, the collision aversion analysis during lane changes showed that the models struggled to capture the physical aspect of human driving: leaving adequate gap between the vehicles. Thus, our analysis highlighted the inadequacy of simple quantitative metrics and the need to take a broader behavioral perspective when analyzing machine-learned models of human driving predictions.