# AEC-Bench: A Multimodal Benchmark for Agentic Systems in Architecture, Engineering, and Construction

arXiv: https://arxiv.org/abs/2603.29199 · HTML: https://mlsanity.com/paper/2603.29199/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 13 of its 20 references that are in the map.
- First posted on arXiv 31 Mar 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 13 of 13)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [Nomic Embed: Training a Reproducible Long Context Text Embedder](https://mlsanity.com/paper/2402.01613/) (2024)
- [MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations](https://mlsanity.com/paper/2407.01523/) (2024)
- [LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating](https://mlsanity.com/paper/2412.18424/) (2024)
- [M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Framework](https://mlsanity.com/paper/2411.06176/) (2024)
- [ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios](https://mlsanity.com/paper/2601.08620/) (2026)
- [AECV-Bench: Benchmarking Multimodal Models on Architectural and Engineering Drawings Understanding](https://mlsanity.com/paper/2601.04819/) (2026)
- [AECBench: A Hierarchical Benchmark for Knowledge Evaluation of Large Language Models in the AEC Field](https://mlsanity.com/paper/2509.18776/) (2025)

## Built on by (most cited first, 6 of 6)

- [StructureClaw: Traceable LLM Agents and an Executable Benchmark for Structural Engineering Workflows](https://mlsanity.com/paper/2607.14896/) (2026)
- [MMArch: Benchmarking Multimodal Reasoning Grounded in Architectural Evidence](https://mlsanity.com/paper/2608.09281/) (2026)
- [Evidence-Grounded Constraint Checking in Construction Documents](https://mlsanity.com/paper/2607.29058/) (2026)
- [CrossProjection: Geometric Grounding Beyond Viewpoint Change in Architectural Drawings](https://mlsanity.com/paper/2608.00473/) (2026)
- [FLOORA: A Human-Aligned Domain-Specific Language Model for Architectural Design](https://mlsanity.com/paper/2609.36064/) (2026)
- [LED-BENCH: A Benchmark for Large-format Engineering Drawing Understanding](https://mlsanity.com/paper/2610.19571/) (2026)

## Abstract (arXiv)

The AEC-Bench is a multimodal benchmark for evaluating agentic systems on real-world tasks in the Architecture, Engineering, and Construction (AEC) domain. The benchmark covers tasks requiring drawing understanding, cross-sheet reasoning, and construction project-level coordination. This report describes the benchmark motivation, dataset taxonomy, evaluation protocol, and baseline results across several domain-specific foundation model harnesses. We use AEC-Bench to identify consistent tools and harness design techniques that uniformly improve performance across foundation models in their own base harnesses, such as Claude Code and Codex. We openly release our benchmark dataset, agent harness, and evaluation code for full replicability at https://github.com/nomic-ai/aec-bench under an Apache 2 license.