# Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements

arXiv: https://arxiv.org/abs/2605.22079 · HTML: https://mlsanity.com/paper/2605.22079/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 29 references that are in the map.
- First posted on arXiv 21 May 2026.

## Builds on (most cited first, 8 of 8)

- [Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning](https://mlsanity.com/paper/2305.13971/) (2023)
- [Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generation](https://mlsanity.com/paper/2403.06988/) (2024)
- [BIM-GPT: a Prompt-Based Virtual Assistant Framework for BIM Information Retrieval](https://mlsanity.com/paper/2304.09333/) (2023)
- [Using Large Language Models for the Interpretation of Building Regulations](https://mlsanity.com/paper/2407.21060/) (2024)
- [ECKGBench: Benchmarking Large Language Models in E-commerce Leveraging Knowledge Graph](https://mlsanity.com/paper/2503.15990/) (2025)
- [EDINET-Bench: Evaluating LLMs on Complex Financial Tasks using Japanese Financial Statements](https://mlsanity.com/paper/2506.08762/) (2025)
- [Qwen-BIM: developing large language model for BIM-based design with domain-specific benchmark and dataset](https://mlsanity.com/paper/2602.20812/) (2026)
- [Building Safer Sites: A Large-Scale Multi-Level Dataset for Construction Safety Research](https://mlsanity.com/paper/2508.09203/) (2025)

## Built on by (most cited first, 1 of 1)

- [Ishigaki-IDS: An Open-Weight Verifier-Aware Model for Information Delivery Specification Drafting in Building Information Modeling](https://mlsanity.com/paper/2606.08545/) (2026)

## Abstract (arXiv)

Building Information Modeling (BIM) projects increasingly use Information Delivery Specification (IDS) to formalize information requirements in a machine-checkable XML format. Because IDS conditions are grounded in the Industry Foundation Classes (IFC) vocabulary, authoring them requires expertise in IFC concepts, validation tools, and property set conventions. Existing benchmarks for structured generation do not adequately capture the additional burden of vocabulary conformance and external-validator agreement that IDS imposes. We present Ishigaki-IDS-Bench, the first publicly released benchmark for IDS generation from BIM information requirements. The benchmark contains 166 examples spanning 83 practical scenarios authored in Japanese and English by six BIM/IDS experts, each paired with a gold IDS file and metadata covering input format, turn setting, target IFC versions, and construction domain. Evaluation proceeds in two stages: (i) formal validity scored by the buildingSMART IDSAuditTool along Processability, Structure, and Content, and (ii) content fidelity scored by facet-level macro-F1 against the gold IDS. Across 10 LLMs in zero-shot, the highest Facet F1 is 65.6%, achieved by GPT-5.5, while the highest Content pass rate is only 33.1%, achieved by Claude Opus 4.5. Ishigaki-IDS-Bench is released on Hugging Face (DOI 10.57967/hf/8873) under CC BY 4.0, and the evaluation code is released on Zenodo (DOI 10.5281/zenodo.20550510) under Apache-2.0.