# Multi-modal Rail Crossing Safety Analysis

arXiv: https://arxiv.org/abs/2607.01365 · HTML: https://mlsanity.com/paper/2607.01365/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 20 references that are in the map.
- First posted on arXiv 1 Jul 2026.

## Builds on (most cited first, 6 of 6)

- [When language and vision meet road safety: leveraging multimodal large language models for video-based traffic accident analysis](https://mlsanity.com/paper/2501.10604/) (2025)
- [ScVLM: Enhancing Vision-Language Model for Safety-Critical Event Understanding](https://mlsanity.com/paper/2410.00982/) (2024)
- [Using Vision Language Models for Safety Hazard Identification in Construction](https://mlsanity.com/paper/2504.09083/) (2025)
- [V-RoAst: Visual Road Assessment. Can VLM be a Road Safety Assessor Using the iRAP Standard?](https://mlsanity.com/paper/2408.10872/) (2024)
- [MonitorVLM:A Vision Language Framework for Safety Violation Detection in Mining Operations](https://mlsanity.com/paper/2510.03666/) (2025)
- [RailVQA: A Benchmark and Framework for Efficient Interpretable Visual Cognition in Automatic Train Operation](https://mlsanity.com/paper/2603.27112/) (2026)

## Abstract (arXiv)

Given one or more images of a railway crossing, can we leverage visual cues that allow us to robustly estimate how safe it is? Can we improve our ability to do so by introducing structured data (such as official accident reports) about the accident history of that crossing into our models? In this work, we explore how to best answer those questions towards building an AI system that can ingest multi-modal data for railway crossings and provide safety assessment and scores that align with expert opinion and with safety scoring used by the Federal Railroad Administration (FRA). To that end, we propose a proof-of-concept pipeline that delivers on that goal, while at the same time exploring and tackling a number of critical research challenges that pertain to different parts of the pipeline, from data preparation to different learning paradigms that can allow us to realize such a system. Indicatively, our proposed system identifies HIGH-RISK and LOW-RISK crossings with a macro F1 score of 0.757 and estimates FRA-based safety scores with an RMSE of 0.078 and correlation of 0.492 using a routed fine-tuned compact VLM pipeline, while producing qualitative results that align with domain-expert assessment.