# Large Language Models in Analyzing Crash Narratives -- A Comparative Study of ChatGPT, BARD and GPT-4

arXiv: https://arxiv.org/abs/2308.13563 · HTML: https://mlsanity.com/paper/2308.13563/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 27 references that are in the map.
- First posted on arXiv 25 Aug 2023.

## Builds on (most cited first, 3 of 3)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)

## Built on by (most cited first, 16 of 16)

- [Exploring Large Language Models for Human Mobility Prediction under Public Events](https://mlsanity.com/paper/2311.17351/) (2023)
- [Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmap](https://mlsanity.com/paper/2503.21411/) (2025)
- [Calibrated Decisions at Scale: Converting Police Crash Narratives into Probabilistic Crash Variables with a System One Model (Jev)](https://mlsanity.com/paper/2609.24052/) (2026)
- [Benchmarking the Capabilities of Large Language Models in Transportation System Engineering: Accuracy, Consistency, and Reasoning Behaviors](https://mlsanity.com/paper/2408.08302/) (2024)
- [Large Language Models and Their Applications in Roadway Safety and Mobility Enhancement: A Comprehensive Review](https://mlsanity.com/paper/2506.06301/) (2025)
- [Large Language Models for Failure Mode Classification: An Investigation](https://mlsanity.com/paper/2309.08181/) (2023)
- [In Generative AI we Trust: Can Chatbots Effectively Verify Political Information?](https://mlsanity.com/paper/2312.13096/) (2023)
- [A New Era in Human Factors Engineering: A Survey of the Applications and Prospects of Large Multimodal Models](https://mlsanity.com/paper/2405.13426/) (2024)
- [TransitGPT: A Generative AI-based framework for interacting with GTFS data using Large Language Models](https://mlsanity.com/paper/2412.06831/) (2024)
- [Enhancing Traffic Prediction with Textual Data Using Large Language Models](https://mlsanity.com/paper/2405.06719/) (2024)
- [Urban Computing in the Era of Large Language Models](https://mlsanity.com/paper/2504.02009/) (2025)
- [An Agentic Workflow for Detecting Personally Identifiable Information in Crash Narratives](https://mlsanity.com/paper/2604.15369/) (2026)
- [Domain-Adapted Pre-trained Language Models for Implicit Information Extraction in Crash Narratives](https://mlsanity.com/paper/2510.09434/) (2025)
- [HARNESS: Human-Agent Risk Navigation and Event Safety System for Proactive Hazard Forecasting in High-Risk DOE Environments](https://mlsanity.com/paper/2511.10810/) (2025)
- [Earth Science Foundation Models: From Perception to Reasoning and Discovery](https://mlsanity.com/paper/2605.12542/) (2026)
- [Large Multimodal Agents for Intelligent Transportation Systems: Architectures, Evidence, and Deployment Challenges](https://mlsanity.com/paper/2608.08184/) (2026)

## Abstract (arXiv)

In traffic safety research, extracting information from crash narratives using text analysis is a common practice. With recent advancements of large language models (LLM), it would be useful to know how the popular LLM interfaces perform in classifying or extracting information from crash narratives. To explore this, our study has used the three most popular publicly available LLM interfaces- ChatGPT, BARD and GPT4. This study investigated their usefulness and boundaries in extracting information and answering queries related to accidents from 100 crash narratives from Iowa and Kansas. During the investigation, their capabilities and limitations were assessed and their responses to the queries were compared. Five questions were asked related to the narratives: 1) Who is at-fault? 2) What is the manner of collision? 3) Has the crash occurred in a work-zone? 4) Did the crash involve pedestrians? and 5) What are the sequence of harmful events in the crash? For questions 1 through 4, the overall similarity among the LLMs were 70%, 35%, 96% and 89%, respectively. The similarities were higher while answering direct questions requiring binary responses and significantly lower for complex questions. To compare the responses to question 5, network diagram and centrality measures were analyzed. The network diagram from the three LLMs were not always similar although they sometimes have the same influencing events with high in-degree, out-degree and betweenness centrality. This study suggests using multiple models to extract viable information from narratives. Also, caution must be practiced while using these interfaces to obtain crucial safety related information.