mlsanityData to 7 Oct 2026

BLUEPRINT Rebuilding a Legacy: Multimodal Retrieval for Complex Engineering Drawings and Documents

arXiv 2602.13345 ↗First posted 12 Feb 2026cs.LG cs.IR cs.MAcited by 0

Not on the map. Why?

Abstract (arXiv)

Decades of engineering drawings and technical records remain locked in legacy archives with inconsistent or missing metadata, making retrieval difficult and often manual. We present Blueprint, a layout-aware multimodal retrieval system designed for large-scale engineering repositories. Blueprint detects canonical drawing regions, applies region-restricted VLM-based OCR, normalizes identifiers (e.g., DWG, part, facility), and fuses lexical and dense retrieval with a lightweight region-level reranker. Deployed on ~770k unlabeled files, it automatically produces structured metadata suitable for cross-facility search. We evaluate Blueprint on a 5k-file benchmark with 350 expert-curated queries using pooled, graded (0/1/2) relevance judgments. Blueprint delivers a 10.1% absolute gain in Success@3 and an 18.9% relative improvement in nDCG@3 over the strongest vision-language baseline}, consistently outperforming across vision, text, and multimodal intents. Oracle ablations reveal substantial headroom under perfect region detection and OCR. We release all queries, runs, annotations, and code to facilitate reproducible evaluation on legacy engineering archives.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 35 of its 72 references that are in the map. First posted on arXiv 12 Feb 2026.

Builds on

35 of 72 references.

References · 35

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2017Billion-scale similarity search with GPUscited by 2,477
  7. 2025Gemma 3 Technical Reportcited by 1,686
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  9. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  10. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  11. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  12. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
23 more
  1. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  2. 2021InfographicVQAcited by 430
  3. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  4. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  5. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  6. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  7. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 345
  8. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  9. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
  10. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  11. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 212
  12. 2024Pixtral 12Bcited by 174
  13. 2022DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysiscited by 146
  14. 2021PubTables-1M: Towards comprehensive table extraction from unstructured documentscited by 122
  15. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Taskscited by 118
  16. 2024LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understandingcited by 94
  17. 2025Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Taskscited by 46
  18. 2018Revisiting the Inverted Indices for Billion-Scale Approximate Nearest Neighborscited by 37
  19. 2024VisionLLaMA: A Unified LLaMA Backbone for Vision Taskscited by 28
  20. 2024Fine-Tuning Vision-Language Model for Automated Engineering Drawing Information Extractioncited by 18
  21. 2025Lost in OCR Translation? Vision-Based Approaches to Robust Document Retrievalcited by 10
  22. 2025From Drawings to Decisions: A Hybrid Vision-Language Framework for Parsing 2D Engineering Drawings into Structured Manufacturing Knowledgecited by 8
  23. 2025A Multi-Stage Hybrid Framework for Automated Interpretation of Multi-View Engineering Drawings Using Vision Language Modelcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026AECV-Bench: Benchmarking Multimodal Models on Architectural and Engineering Drawings Understanding
  2. 2026Text-to-CAD Retrieval: a Strong Baseline
  3. 2025MMDocIR: Benchmarking Multimodal Retrieval for Long Documents
  4. 2026MMArch: Benchmarking Multimodal Reasoning Grounded in Architectural Evidence
  5. 2026ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge
  6. 2026Overview of the EReL@MIR 2025 Multimodal Document Retrieval Challenge (Track 1)
  7. 2025Beyond Patch Aggregation: 3-Pass Pyramid Indexing for Vision-Enhanced Document Retrieval
  8. 2026IRPAPERS: A Visual Document Benchmark for Scientific Retrieval and Question Answering