# Benchmarking Frontier Large Language Models Against Official Crash Database Coding Using Police Crash Narratives

arXiv: https://arxiv.org/abs/2607.29064 · HTML: https://mlsanity.com/paper/2607.29064/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 30 references that are in the map.
- First posted on arXiv 31 Jul 2026.

## Builds on (most cited first, 5 of 5)

- [Large Language Models and Their Applications in Roadway Safety and Mobility Enhancement: A Comprehensive Review](https://mlsanity.com/paper/2506.06301/) (2025)
- [Exploring Traffic Crash Narratives in Jordan Using Text Mining Analytics](https://mlsanity.com/paper/2406.09438/) (2024)
- [Large Language Models for Traffic and Transportation Research: Methodologies, State of the Art, and Future Opportunities](https://mlsanity.com/paper/2503.21330/) (2025)
- [Predicting person-level injury severity using crash narratives: A balanced approach with roadway classification and natural language process techniques](https://mlsanity.com/paper/2509.07845/) (2025)
- [Towards Reliable and Interpretable Traffic Crash Pattern Prediction and Safety Interventions Using Customized Large Language Models](https://mlsanity.com/paper/2505.12545/) (2025)

## Built on by (most cited first, 2 of 2)

- [Calibrated Decisions at Scale: Converting Police Crash Narratives into Probabilistic Crash Variables with a System One Model (Jev)](https://mlsanity.com/paper/2609.24052/) (2026)
- [Estimating Uncoded Crash Factors with Tabular Foundation and System One Models: Kumo Tabular and Jev](https://mlsanity.com/paper/2610.10321/) (2026)

## Abstract (arXiv)

Police crash narratives contain information that may supplement structured crash databases, but manual review is labor-intensive and it remains unclear how well large language models (LLMs) reproduce official crash coding. This study benchmarked six frontier LLMs by comparing narrative-derived crash attribute codes with corresponding fields in the Arkansas fatal-crash database. The analysis linked 5,587 fatal-crash narratives with 5,889 structured crash records from Arkansas (2015-2025), yielding 4,194 matched crashes. Six LLMs were evaluated using an identical zero-shot prompt to code crash manner, non-motorist relation, intersection type, work-zone relation, roadway surface condition, and light condition. Performance was evaluated using agreement, macro-averaged F1 score, Cohen's kappa, coverage, selective agreement, and comparisons with always-majority, always-Unknown, and keyword-rule baselines. Repeated-measures analyses and a generalized estimating equations model assessed differences among models and attributes. GPT-5.5 High achieved the highest agreement among the evaluated LLMs, but the always-majority baseline produced higher raw agreement and the keyword-rule baseline achieved macro-averaged F1 score and Cohen's kappa comparable to the best-performing LLM. Agreement was highest for non-motorist relation and crash manner and lowest for light condition, roadway surface condition, and work-zone relation. Differences across crash attributes exceeded differences across models. These results provide a benchmark for evaluating LLM-based crash coding and show that deployment should be evaluated on an attribute-specific basis using transparent baselines and human review.