# Domain-Adapted Pre-trained Language Models for Implicit Information Extraction in Crash Narratives

arXiv: https://arxiv.org/abs/2510.09434 · HTML: https://mlsanity.com/paper/2510.09434/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 55 references that are in the map.
- First posted on arXiv 10 Oct 2025.

## Builds on (most cited first, 17 of 17)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Qwen2.5-1M Technical Report](https://mlsanity.com/paper/2501.15383/) (2025)
- [Recurrent Neural Network for Text Classification with Multi-Task Learning](https://mlsanity.com/paper/1605.05101/) (2016)
- [A Survey on LoRA of Large Language Models](https://mlsanity.com/paper/2407.11046/) (2024)
- [On a generalization of the Jensen-Shannon divergence and the JS-symmetrization of distances relying on abstract means](https://mlsanity.com/paper/1904.04017/) (2019)
- [Large Language Models in Analyzing Crash Narratives -- A Comparative Study of ChatGPT, BARD and GPT-4](https://mlsanity.com/paper/2308.13563/) (2023)
- [Selective Prompt Anchoring for Code Generation](https://mlsanity.com/paper/2408.09121/) (2024)
- [Leveraging Large Language Models with Chain-of-Thought and Prompt Engineering for Traffic Crash Severity Analysis and Inference](https://mlsanity.com/paper/2408.04652/) (2024)
- [iQUEST: An Iterative Question-Guided Framework for Knowledge Base Question Answering](https://mlsanity.com/paper/2506.01784/) (2025)
- [Exploring Traffic Crash Narratives in Jordan Using Text Mining Analytics](https://mlsanity.com/paper/2406.09438/) (2024)
- [Safety in Traffic Management Systems: A Comprehensive Survey](https://mlsanity.com/paper/2308.06204/) (2023)
- [Analysis of Railway Accidents' Narratives Using Deep Learning](https://mlsanity.com/paper/1810.07382/) (2018)
- [LLM driven Text-to-Table Generation through Sub-Tasks Guidance and Iterative Refinement](https://mlsanity.com/paper/2508.08653/) (2025)

## Built on by (most cited first, 1 of 1)

- [LLM-Powered Workflow Optimization for Multidisciplinary Software Development: An Automotive Industry Case Study](https://mlsanity.com/paper/2603.21439/) (2026)

## Abstract (arXiv)

Free-text crash narratives recorded in real-world crash databases have been shown to play a significant role in improving traffic safety. However, large-scale analyses remain difficult to implement as there are no documented tools that can batch process the unstructured, non standardized text content written by various authors with diverse experience and attention to detail. In recent years, Transformer-based pre-trained language models (PLMs), such as Bidirectional Encoder Representations from Transformers (BERT) and large language models (LLMs), have demonstrated strong capabilities across various natural language processing tasks. These models can extract explicit facts from crash narratives, but their performance declines on inference-heavy tasks in, for example, Crash Type identification, which can involve nearly 100 categories. Moreover, relying on closed LLMs through external APIs raises privacy concerns for sensitive crash data. Additionally, these black-box tools often underperform due to limited domain knowledge. Motivated by these challenges, we study whether compact open-source PLMs can support reasoning-intensive extraction from crash narratives. We target two challenging objectives: 1) identifying the Manner of Collision for a crash, and 2) Crash Type for each vehicle involved in the crash event from real-world crash narratives. To bridge domain gaps, we apply fine-tuning techniques to inject task-specific knowledge to LLMs with Low-Rank Adaption (LoRA) and BERT. Experiments on the authoritative real-world dataset Crash Investigation Sampling System (CISS) demonstrate that our fine-tuned compact models outperform strong closed LLMs, such as GPT-4o, while requiring only minimal training resources. Further analysis reveals that the fine-tuned PLMs can capture richer narrative details and even correct some mislabeled annotations in the dataset.