# AECV-Bench: Benchmarking Multimodal Models on Architectural and Engineering Drawings Understanding

arXiv: https://arxiv.org/abs/2601.04819 · HTML: https://mlsanity.com/paper/2601.04819/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 0 of its 1 references that are in the map.
- First posted on arXiv 8 Jan 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Built on by (most cited first, 11 of 11)

- [AEC-Bench: A Multimodal Benchmark for Agentic Systems in Architecture, Engineering, and Construction](https://mlsanity.com/paper/2603.29199/) (2026)
- [MechVQA: Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understanding](https://mlsanity.com/paper/2605.30794/) (2026)
- [Engineering Reasoning and Instruction (ERI) Benchmark: A Large Taxonomy-driven Dataset for Foundation Models and Agents](https://mlsanity.com/paper/2603.02239/) (2026)
- [ArchSIBench: Benchmarking the Architectural Spatial Intelligence of Vision-Language Models](https://mlsanity.com/paper/2605.20837/) (2026)
- [StructureClaw: Traceable LLM Agents and an Executable Benchmark for Structural Engineering Workflows](https://mlsanity.com/paper/2607.14896/) (2026)
- [MMArch: Benchmarking Multimodal Reasoning Grounded in Architectural Evidence](https://mlsanity.com/paper/2608.09281/) (2026)
- [CrossProjection: Geometric Grounding Beyond Viewpoint Change in Architectural Drawings](https://mlsanity.com/paper/2608.00473/) (2026)
- [ARCH-B: Architectural Representation, Comprehension and Hierarchy Benchmark](https://mlsanity.com/paper/2609.34047/) (2026)
- [FLOORA: A Human-Aligned Domain-Specific Language Model for Architectural Design](https://mlsanity.com/paper/2609.36064/) (2026)
- [Why VLMs Miss Small Objects, and When Zooming In Is Safe](https://mlsanity.com/paper/2610.09313/) (2026)
- [LED-BENCH: A Benchmark for Large-format Engineering Drawing Understanding](https://mlsanity.com/paper/2610.19571/) (2026)

## Abstract (arXiv)

AEC drawings encode geometry and semantics through symbols, layout conventions, and dense annotation, yet it remains unclear whether modern multimodal and vision-language models can reliably interpret this graphical language. We present AECV-Bench, a benchmark for evaluating multimodal and vision-language models on realistic AEC artefacts via two complementary use cases: (i) object counting on 120 high-quality floor plans (doors, windows, bedrooms, toilets), and (ii) drawing-grounded document QA spanning 192 question-answer pairs that test text extraction (OCR), instance counting, spatial reasoning, and comparative reasoning over common drawing regions. Object-counting performance is reported using per-field exact-match accuracy and MAPE results, while document-QA performance is reported using overall accuracy and per-category breakdowns with an LLM-as-a-judge scoring pipeline and targeted human adjudication for edge cases. Evaluating a broad set of state-of-the-art models under a unified protocol, we observe a stable capability gradient; OCR and text-centric document QA are strongest (up to 0.95 accuracy), spatial reasoning is moderate, and symbol-centric drawing understanding - especially reliable counting of doors and windows - remains unsolved (often 0.40-0.55 accuracy) with substantial proportional errors. These results suggest that current systems function well as document assistants but lack robust drawing literacy, motivating domain-specific representations and tool-augmented, human-in-the-loop workflows for an efficient AEC automation.