# LLM-based HSE Compliance Assessment: Benchmark, Performance, and Advancements

arXiv: https://arxiv.org/abs/2505.22959 · HTML: https://mlsanity.com/paper/2505.22959/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 24 of its 69 references that are in the map.
- First posted on arXiv 29 May 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models](https://mlsanity.com/paper/2305.04091/) (2023)
- [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://mlsanity.com/paper/2308.11462/) (2023)
- [Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning](https://mlsanity.com/paper/2403.16999/) (2024)
- [SafetyBench: Evaluating the Safety of Large Language Models](https://mlsanity.com/paper/2309.07045/) (2023)
- [Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models](https://mlsanity.com/paper/2401.01301/) (2024)
- [FinBen: A Holistic Financial Benchmark for Large Language Models](https://mlsanity.com/paper/2402.12659/) (2024)
- [Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation](https://mlsanity.com/paper/2309.17234/) (2023)
- [SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers](https://mlsanity.com/paper/2407.09413/) (2024)
- [MedCalc-Bench: Evaluating Large Language Models for Medical Calculations](https://mlsanity.com/paper/2406.12036/) (2024)
- [LabSafety Bench: Benchmarking LLMs on Safety Issues in Scientific Labs](https://mlsanity.com/paper/2410.14182/) (2024)
- [On LLM-Enhanced Mixed-Type Data Imputation with High-Order Message Passing](https://mlsanity.com/paper/2501.02191/) (2025)
- [Efficient Unsupervised Community Search with Pre-trained Graph Transformer](https://mlsanity.com/paper/2403.18869/) (2024)
- [Vision Language Model for Interpretable and Fine-grained Detection of Safety Compliance in Diverse Workplaces](https://mlsanity.com/paper/2408.07146/) (2024)
- [BEACON: Benchmark for Comprehensive RNA Tasks and Language Models](https://mlsanity.com/paper/2406.10391/) (2024)
- [Synthetic Context Generation for Question Generation](https://mlsanity.com/paper/2406.13188/) (2024)
- [Summarizing long regulatory documents with a multi-step pipeline](https://mlsanity.com/paper/2408.09777/) (2024)
- [The Power of Stories: Narrative Priming Shapes How LLM Agents Collaborate and Compete](https://mlsanity.com/paper/2505.03961/) (2025)

## Built on by (most cited first, 9 of 9)

- [AutoMonitor-Bench: Evaluating the Reliability of LLM-Based Misbehavior Monitor](https://mlsanity.com/paper/2601.05752/) (2026)
- [AIReg-Bench: Benchmarking Language Models That Assess AI Regulation Compliance](https://mlsanity.com/paper/2510.01474/) (2025)
- [PRoH: Dynamic Planning and Reasoning over Knowledge Hypergraphs for Retrieval-Augmented Generation](https://mlsanity.com/paper/2510.12434/) (2025)
- [COMPASS: A Framework for Evaluating Organization-Specific Policy Alignment in LLMs](https://mlsanity.com/paper/2601.01836/) (2026)
- [Automated Facility Enumeration for Building Compliance Checking using Door Detection and Large Language Models](https://mlsanity.com/paper/2509.17283/) (2025)
- [Evaluating Implicit Regulatory Compliance in LLM Tool Invocation via Logic-Guided Synthesis](https://mlsanity.com/paper/2601.08196/) (2026)
- [WuYu-EnvLE-Bench: A Benchmark for Evaluating Large Language Models in Environmental Law Enforcement](https://mlsanity.com/paper/2607.17745/) (2026)
- [Interpretable Column Annotation with LLM-Symbolized Decision Process Materialization](https://mlsanity.com/paper/2607.25228/) (2026)
- [Calibrating Post-Training Feature Shifts for LLM Data Contamination Detection](https://mlsanity.com/paper/2608.10462/) (2026)

## Abstract (arXiv)

Health, Safety, and Environment (HSE) compliance assessment demands dynamic real-time decision-making under complicated regulations and complex human-machine-environment interactions. While large language models (LLMs) hold significant potential for decision intelligence and contextual dialogue, their capacity for domain-specific knowledge in HSE and structured legal reasoning remains underexplored. We introduce HSE-Bench, the first benchmark dataset designed to evaluate the HSE compliance assessment capabilities of LLM. HSE-Bench comprises over 1,000 manually curated questions drawn from regulations, court cases, safety exams, and fieldwork videos, and integrates a reasoning flow based on Issue spotting, rule Recall, rule Application, and rule Conclusion (IRAC) to assess the holistic reasoning pipeline. We conduct extensive evaluations on different prompting strategies and more than 10 LLMs, including foundation models, reasoning models and multimodal vision models. The results show that, although current LLMs achieve good performance, their capabilities largely rely on semantic matching rather than principled reasoning grounded in the underlying HSE compliance context. Moreover, their native reasoning trace lacks the systematic legal reasoning required for rigorous HSE compliance assessment. To alleviate these, we propose a new prompting technique, Reasoning of Expert (RoE), which guides LLMs to simulate the reasoning process of different experts for compliance assessment and reach a more accurate unified decision. We hope our study highlights reasoning gaps in LLMs for HSE compliance and inspires further research on related tasks.