# SAFARI: An Industrial Benchmark for LLM-Assisted Hazard Analysis and Risk Assessment

arXiv: https://arxiv.org/abs/2609.20584 · HTML: https://mlsanity.com/paper/2609.20584/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 28 references that are in the map.
- First posted on arXiv 17 Sep 2026.

## Builds on (most cited first, 17 of 17)

- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [Scenarios for Development, Test and Validation of Automated Vehicles](https://mlsanity.com/paper/1801.08598/) (2018)
- [Ontology based Scene Creation for the Development of Automated Vehicles](https://mlsanity.com/paper/1704.01006/) (2017)
- [Do LLM Evaluators Prefer Themselves for a Reason?](https://mlsanity.com/paper/2504.03846/) (2025)
- [When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Tasks](https://mlsanity.com/paper/2311.08993/) (2023)
- [Safety Analysis in the Era of Large Language Models: A Case Study of STPA using ChatGPT](https://mlsanity.com/paper/2304.01246/) (2023)
- [Lost in Pronunciation: Detecting Chinese Offensive Language Disguised by Phonetic Cloaking Replacement](https://mlsanity.com/paper/2507.07640/) (2025)
- [Can GRPO Boost Complex Multimodal Table Understanding?](https://mlsanity.com/paper/2509.16889/) (2025)
- [Milestones in Autonomous Driving and Intelligent Vehicles Part II: Perception and Planning](https://mlsanity.com/paper/2306.01980/) (2023)
- [Aegis:An Advanced LLM-Based Multi-Agent for Intelligent Functional Safety Engineering](https://mlsanity.com/paper/2410.12475/) (2024)
- [MedFact: A Large-scale Chinese Dataset for Evidence-based Medical Fact-checking of LLM Responses](https://mlsanity.com/paper/2509.17436/) (2025)
- [PsychEthicsBench: Evaluating Large Language Models Against Australian Mental Health Ethics](https://mlsanity.com/paper/2601.03578/) (2026)
- [Do No Harm: Exposing Hidden Vulnerabilities of LLMs via Persona-based Client Simulation Attack in Psychological Counseling](https://mlsanity.com/paper/2604.04842/) (2026)
- [Template-Driven LLM-Paraphrased Framework for Tabular Math Word Problem Generation](https://mlsanity.com/paper/2412.15594/) (2024)
- [Towards Efficient Hazard Identification in the Concept Phase of Driverless Vehicle Development](https://mlsanity.com/paper/2004.10501/) (2020)
- [Identification of Potential Hazardous Events for an Unmanned Protective Vehicle](https://mlsanity.com/paper/1804.08728/) (2018)
- [AudioProcessBench: Benchmark for Identifying Process Errors in Audio-Grounded Reasoning](https://mlsanity.com/paper/2606.09925/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly considered for safety-critical engineering, yet their reliability in regulated functional-safety workflows remains underexplored. We introduce SAFARI (Safety-Aware Functional Automotive Risk Inference), the first industrial benchmark for LLM-assisted automotive Hazard Analysis and Risk Assessment (HARA) under ISO 26262. It contains 3,000 de-identified industrial HARA cases and evaluates two coupled tasks: open-ended hazard analysis and standards-grounded risk assessment. To evaluate open-ended HARA artifacts, we propose the first reference-anchored LLM-as-a-judge protocol with high expert correlation. Experiments with nine frontier LLMs show that models often produce plausible hazard narratives but remain weak at ISO 26262 risk classification, with the best ASIL macro-F1 reaching only 0.261. Chain-of-Thought prompting provides limited benefit and often degrades categorical risk assessment. Error analysis further localizes major failures to scenario-critical context omissions during hazard generation and to controllability misjudgments during risk assessment, indicating where expert oversight should be concentrated. The dataset can be obtained from https://github.com/xixi47520-hash/HARA.