# PSI: A Benchmark for Human Interpretation and Response in Traffic Interactions

arXiv: https://arxiv.org/abs/2112.02604 · HTML: https://mlsanity.com/paper/2112.02604/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 75 references that are in the map.
- First posted on arXiv 5 Dec 2021.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021)
- [Textual Explanations for Self-Driving Vehicles](https://mlsanity.com/paper/1807.11546/) (2018)
- [End-to-end Learning of Driving Models from Large-scale Video Datasets](https://mlsanity.com/paper/1612.01079/) (2016)
- [Visual Compositional Learning for Human-Object Interaction Detection](https://mlsanity.com/paper/2007.12407/) (2020)
- [Explainable Object-induced Action Decision for Autonomous Vehicles](https://mlsanity.com/paper/2003.09405/) (2020)
- [Grounding Human-to-Vehicle Advice for Self-driving Vehicles](https://mlsanity.com/paper/1911.06978/) (2019)
- [Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction](https://mlsanity.com/paper/2002.08945/) (2020)
- [Stepwise Goal-Driven Networks for Trajectory Prediction](https://mlsanity.com/paper/2103.14107/) (2021)
- [TITAN: Future Forecast using Action Priors](https://mlsanity.com/paper/2003.13886/) (2020)
- [LOKI: Long Term and Key Intentions for Trajectory Prediction](https://mlsanity.com/paper/2108.08236/) (2021)
- [Predicting Pedestrian Crossing Intention with Feature Fusion and Spatio-Temporal Attention](https://mlsanity.com/paper/2104.05485/) (2021)
- [BiTraP: Bi-directional Pedestrian Trajectory Prediction with Multi-modal Goal Estimation](https://mlsanity.com/paper/2007.14558/) (2020)
- [JRDB-Act: A Large-scale Dataset for Spatio-temporal Action, Social Group and Activity Detection](https://mlsanity.com/paper/2106.08827/) (2021)
- [Joint Attention in Autonomous Driving (JAAD)](https://mlsanity.com/paper/1609.04741/) (2016)
- [PedX: Benchmark Dataset for Metric 3D Pose Estimation of Pedestrians in Complex Urban Intersections](https://mlsanity.com/paper/1809.03605/) (2018)
- [Euro-PVI: Pedestrian Vehicle Interactions in Dense Urban Centers](https://mlsanity.com/paper/2106.12442/) (2021)
- [Pedestrian Behavior Prediction for Automated Driving: Requirements, Metrics, and Relevant Features](https://mlsanity.com/paper/2012.08418/) (2020)
- [Attentional-GCNN: Adaptive Pedestrian Trajectory Prediction towards Generic Autonomous Vehicle Use Cases](https://mlsanity.com/paper/2011.11190/) (2020)
- [AutonoVi: Autonomous Vehicle Planning with Dynamic Maneuvers and Traffic Constraints](https://mlsanity.com/paper/1703.08561/) (2017)

## Built on by (most cited first, 17 of 17)

- [Explainable AI for Safe and Trustworthy Autonomous Driving: A Systematic Review](https://mlsanity.com/paper/2402.10086/) (2024)
- [Behavioral Intention Prediction in Driving Scenes: A Survey](https://mlsanity.com/paper/2211.00385/) (2022)
- [RAG-based Explainable Prediction of Road Users Behaviors for Automated Driving using Knowledge Graphs and Large Language Models](https://mlsanity.com/paper/2405.00449/) (2024)
- [Large Language Models and Their Applications in Roadway Safety and Mobility Enhancement: A Comprehensive Review](https://mlsanity.com/paper/2506.06301/) (2025)
- [Learning Pedestrian Actions to Ensure Safe Autonomous Driving](https://mlsanity.com/paper/2305.13051/) (2023)
- [A Survey on Datasets for Decision-making of Autonomous Vehicle](https://mlsanity.com/paper/2306.16784/) (2023)
- [Diving Deeper Into Pedestrian Behavior Understanding: Intention Estimation, Action Prediction, and Event Risk Assessment](https://mlsanity.com/paper/2407.00446/) (2024)
- [Can Reasons Help Improve Pedestrian Intent Estimation? A Cross-Modal Approach](https://mlsanity.com/paper/2411.13302/) (2024)
- [Pedestrian Motion Prediction Using Transformer-based Behavior Clustering and Data-Driven Reachability Analysis](https://mlsanity.com/paper/2408.15250/) (2024)
- [Online Residual Learning from Offline Experts for Pedestrian Tracking](https://mlsanity.com/paper/2409.04069/) (2024)
- [Gating Syn-to-Real Knowledge for Pedestrian Crossing Prediction in Safe Driving](https://mlsanity.com/paper/2409.06707/) (2024)
- [Pedestrian Intention and Trajectory Prediction in Unstructured Traffic Using IDD-PeD](https://mlsanity.com/paper/2506.22111/) (2025)
- [VLMs Guided Interpretable Decision Making for Autonomous Driving](https://mlsanity.com/paper/2511.13881/) (2025)
- [From Detection to Understanding: TAR and TAR-Bench for Multi-Task Traffic Anomaly Reasoning](https://mlsanity.com/paper/2608.10317/) (2026)
- [A low complexity contextual stacked ensemble-learning approach for pedestrian intent prediction](https://mlsanity.com/paper/2410.13039/) (2024)
- [COSI-Lab: Conference Living Lab for Modeling Multi-Perspective Multimodal Social Intention](https://mlsanity.com/paper/2607.28649/) (2026)
- [TAU-Agent: An Agentic Retrieval-Augmented Framework for Traffic Anomaly Understanding](https://mlsanity.com/paper/2608.25935/) (2026)

## Abstract (arXiv)

Accurately modeling pedestrian intention and understanding driver decision-making processes are critical for the development of safe and socially aware autonomous driving systems. We introduce PSI, a benchmark dataset that captures the dynamic evolution of pedestrian crossing intentions from the driver's perspective, enriched with human textual explanations that reflect the reasoning behind intention estimation and driving decision making. These annotations offer a unique foundation for developing and benchmarking models that combine predictive performance with interpretable and human-aligned reasoning. PSI supports standardized tasks and evaluation protocols across multiple dimensions, including pedestrian intention prediction, driver decision modeling, reasoning generation, and trajectory forecasting and more. By enabling causal and interpretable evaluation, PSI advances research toward autonomous systems that can reason, act, and explain in alignment with human cognitive processes.