# Towards a Benchmark for Fire Safety Code Interpretation in the Hong Kong AEC Industry

arXiv: https://arxiv.org/abs/2610.16127 · HTML: https://mlsanity.com/paper/2610.16127/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 3 of its 27 references that are in the map.
- First posted on arXiv 12 Oct 2026.

## Builds on (most cited first, 3 of 3)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Toward Engineering AGI: Benchmarking the Engineering Design Capabilities of LLMs](https://mlsanity.com/paper/2509.16204/) (2025)
- [Automating construction contract review using knowledge graph-enhanced large language models](https://mlsanity.com/paper/2309.12132/) (2023)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly explored in the Architecture, Engineering, and Construction (AEC) domain, yet their reliability in regulatory compliance remains uncertain. In this study, we evaluate the performance of LLMs on Hong Kong fire safety codes. In collaboration with domain experts, we create a localized benchmark dataset comprising both multiple-choice questions (MCQs) and open-ended questions (OEQs) to evaluate 12 widely used proprietary and open-source models. Our findings show that top-performing proprietary models achieve 80-90% accuracy on MCQs, whereas open-source models generally lag behind. However, on open-ended evaluation, expert human assessments reveal an accuracy range of only 50-70%, demonstrating that LLMs still have substantial room for improvement in specialized compliance tasks.