# Enginuity: A Dataset and Benchmark for Vision-Language Understanding of Engineering Diagrams

arXiv: https://arxiv.org/abs/2606.03410 · HTML: https://mlsanity.com/paper/2606.03410/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 2 Jun 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 17 of 17)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [A Diagram Is Worth A Dozen Images](https://mlsanity.com/paper/1603.07396/) (2016)
- [Quantifying the Carbon Emissions of Machine Learning](https://mlsanity.com/paper/1910.09700/) (2019)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022)
- [Document Understanding Dataset and Evaluation (DUDE)](https://mlsanity.com/paper/2305.08455/) (2023)
- [SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoning](https://mlsanity.com/paper/2505.19099/) (2025)
- [ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models](https://mlsanity.com/paper/2505.13444/) (2025)
- [FlowVQA: Mapping Multimodal Logic in Visual Question Answering with Flowcharts](https://mlsanity.com/paper/2406.19237/) (2024)
- [IndustryEQA: Pushing the Frontiers of Embodied Question Answering in Industrial Scenarios](https://mlsanity.com/paper/2505.20640/) (2025)
- [An Automated Engineering Assistant: Learning Parsers for Technical Drawings](https://mlsanity.com/paper/1909.08552/) (2019)
- [Enginuity: Building an Open Multi-Domain Dataset of Complex Engineering Diagrams](https://mlsanity.com/paper/2601.13299/) (2026)

## Built on by (most cited first, 1 of 1)

- [Knossos and Ariadne: Benchmarking and Learning Complete Diagram Topology Extraction with Vision-Language Models](https://mlsanity.com/paper/2610.04721/) (2026)

## Abstract (arXiv)

Engineering diagrams pose a distinct challenge for vision-language models: unlike natural images or general documents, they encode information through dense spatial layouts, domain-specific symbols, and cross-references between visual callouts and structured parts tables. Despite their centrality to service, repair, and design workflows, there is no public benchmark for measuring VLM capabilities in this domain; existing datasets primarily focus on flowcharts, scientific figures, or business documents. To address this gap, we introduce Enginuity, the first open dataset and benchmark for evaluating VLMs on complex engineering diagrams. We define two tasks over a corpus of U.S. military service and repair manuals: structured parts-table extraction (Task 1) and free-form visual diagram question answering (VQA)(Task 2) for benchmarking. We evaluate four frontier VLMs (GPT-5.2 Chat, Claude Opus 4.7, Gemma 4, Qwen3-VL-32B-Instruct) under zero-shot and chain-of-thought prompting. On Task 1, models reach Recall@all of 0.61-0.87 but Token F1pen of only 0.03-0.18, exposing a systematic gap between part identification and description fidelity. Task 2 reveals a consistent factual-reasoning gap across all models. A supporting analysis shows that token-overlap metrics under-report model capability on technical descriptions by 2-6x relative to semantic similarity, motivating LLM-as-judge calibration for domain-specific evaluation. We release the dataset, annotations, evaluation harness, and per-sample model outputs to support a reproducible study of VLM capability on engineering content.