mlsanityData to 7 Oct 2026

AEC-Bench: A Multimodal Benchmark for Agentic Systems in Architecture, Engineering, and Construction

arXiv 2603.29199 ↗First posted 31 Mar 2026cs.AIcited by 4

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

The AEC-Bench is a multimodal benchmark for evaluating agentic systems on real-world tasks in the Architecture, Engineering, and Construction (AEC) domain. The benchmark covers tasks requiring drawing understanding, cross-sheet reasoning, and construction project-level coordination. This report describes the benchmark motivation, dataset taxonomy, evaluation protocol, and baseline results across several domain-specific foundation model harnesses. We use AEC-Bench to identify consistent tools and harness design techniques that uniformly improve performance across foundation models in their own base harnesses, such as Claude Code and Codex. We openly release our benchmark dataset, agent harness, and evaluation code for full replicability at https://github.com/nomic-ai/aec-bench under an Apache 2 license.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 13 of its 20 references that are in the map. First posted on arXiv 31 Mar 2026.

Builds on

13 of 20 references.

References · 13

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  3. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  4. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  5. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  6. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  7. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255
  8. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  9. 2024LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locatingcited by 70
  10. 2024M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Frameworkcited by 36
  11. 2026ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarioscited by 22
  12. 2026AECV-Bench: Benchmarking Multimodal Models on Architectural and Engineering Drawings Understandingcited by 8
1 more
  1. 2025AECBench: A Hierarchical Benchmark for Knowledge Evaluation of Large Language Models in the AEC Fieldcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026MMArch: Benchmarking Multimodal Reasoning Grounded in Architectural Evidencecited by 1
  2. 2026StructureClaw: Traceable LLM Agents and an Executable Benchmark for Structural Engineering Workflowscited by 0
  3. 2026Evidence-Grounded Constraint Checking in Construction Documentscited by 0
  4. 2026CrossProjection: Geometric Grounding Beyond Viewpoint Change in Architectural Drawingscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarks
  2. 2025REALM-Bench: A Benchmark for Evaluating Multi-Agent Systems on Real-world, Dynamic Planning and Scheduling Tasks
  3. 2026Embodied-BenchForge: A Closed-Loop Agentic Workflow for Embodied Benchmark Construction
  4. 2026AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation
  5. 2025Automating Benchmark Design
  6. 2026Benchmark Everything Everywhere All at Once
  7. 2026ELT-Bench-Verified: Benchmark Quality Issues Underestimate AI Agent Capabilities
  8. 2026Is Agentic AI Ready for Real-World Hardware Engineering? A Deep Dive with Phoenix-bench