mlsanityData to 14 Oct 2026

From Concept to Manufacturing: Evaluating Vision-Language Models for Engineering Design

arXiv 2311.12668 ↗First posted 21 Nov 2023cs.AI cs.CEcited by 51

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Engineering design is undergoing a transformative shift with the advent of AI, marking a new era in how we approach product, system, and service planning. Large language models have demonstrated impressive capabilities in enabling this shift. Yet, with text as their only input modality, they cannot leverage the large body of visual artifacts that engineers have used for centuries and are accustomed to. This gap is addressed with the release of multimodal vision-language models (VLMs), such as GPT-4V, enabling AI to impact many more types of tasks. Our work presents a comprehensive evaluation of VLMs across a spectrum of engineering design tasks, categorized into four main areas: Conceptual Design, System-Level and Detailed Design, Manufacturing and Inspection, and Engineering Education Tasks. Specifically in this paper, we assess the capabilities of two VLMs, GPT-4V and LLaVA 1.6 34B, in design tasks such as sketch similarity analysis, CAD generation, topology optimization, manufacturability assessment, and engineering textbook problems. Through this structured evaluation, we not only explore VLMs' proficiency in handling complex design challenges but also identify their limitations in complex engineering design applications. Our research establishes a foundation for future assessments of vision language models. It also contributes a set of benchmark testing datasets, with more than 1000 queries, for ongoing advancements and applications in this field.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 103 references that are in the map. First posted on arXiv 21 Nov 2023.

Builds on

16 of 103 references.

References · 16

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  3. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  4. 2022A Survey on In-context Learningcited by 721
  5. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  6. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
  7. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  8. 2020TopologyGAN: Topology Optimization Using Generative Adversarial Networks Based on Physical Fields Over the Initial Domaincited by 65
  9. 2023Aligning Optimization Trajectories with Diffusion Models for Constrained Design Generationcited by 52
  10. 2022On the use of Artificial Neural Networks in Topology Optimisationcited by 36
  11. 2023MeLM, a generative pretrained language modeling framework that solves forward and inverse mechanics problemscited by 29
  12. 2023Multi-modal Machine Learning in Engineering Design: A Review and Future Directionscited by 27
4 more
  1. 2019Meta-learning Convolutional Neural Architectures for Multi-target Concrete Defect Classification with the COncrete DEfect BRidge IMage Datasetcited by 21
  2. 2023GPT Models in Construction Industry: Opportunities, Limitations, and a Use Case Validationcited by 21
  3. 2021Natural Language Processing in-and-for Design Researchcited by 15
  4. 2023Multi-modal Machine Learning for Vehicle Rating Predictions Using Image, Text, and Parametric Datacited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 16; 2025: 17; 2026: 18.

51 papers.

Sort:

By area:

  1. 2024DesignQA: A Multimodal Benchmark for Evaluating Large Language Models' Understanding of Engineering Documentationcited by 32
  2. 2024Large Language Models for Manufacturingcited by 24
  3. 2024Large Language Model Agent as a Mechanical Designercited by 21
  4. 2025Literature-Grounded Novelty Assessment of Scientific Ideascited by 19
  5. 2024LLM-3D Print: Large Language Models To Monitor and Control 3D Printingcited by 18
  6. 2025Large Language Models for Computer-Aided Design: A Surveycited by 18
  7. 2025From Idea to CAD: A Language Model-Driven Multi-Agent System for Collaborative Designcited by 14
  8. 2024Sketch2Prototype: Rapid Conceptual Design Exploration and Prototyping with Generative AIcited by 13
  9. 2024Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptionscited by 13
  10. 2026CADBench: A Multimodal Benchmark for AI-Assisted CAD Program Generationcited by 11
  11. 2026Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideascited by 10
  12. 2024AutoTRIZ: Automating Engineering Innovation with TRIZ and Large Language Modelscited by 9
38 more
  1. 2025SoM-1K: A Thousand-Problem Benchmark Dataset for Strength of Materialscited by 8
  2. 2025Agentic Large Language Models for Conceptual Systems Engineering and Designcited by 7
  3. 2026Distilled Large Language Model-Driven Dynamic Sparse Expert Activation Mechanismcited by 7
  4. 2025EvoCAD: Evolutionary CAD Code Generation with Vision Language Modelscited by 6
  5. 2024From Cloud to Edge: Rethinking Generative AI for Low-Resource Design Challengescited by 5
  6. 2025Toward Autonomous Engineering Design: A Knowledge-Guided Multi-Agent Frameworkcited by 5
  7. 2024Industrial Language-Image Dataset (ILID): Adapting Vision Foundation Models for Industrial Settingscited by 4
  8. 2026MechVQA: Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understandingcited by 4
  9. 2024Exploring the Potentials and Challenges of Deep Generative Models in Product Design Conceptioncited by 3
  10. 2025Towards an automated workflow in materials science for combining multi-modal simulative and experimental information using data mining and large language modelscited by 3
  11. 2025GRAID: Enhancing Spatial Reasoning of VLMs Through High-Fidelity Data Generationcited by 3
  12. 2024Evaluating Large Language Models for Material Selectioncited by 2
  13. 2024GeoBiked: A Dataset with Geometric Features and Automated Labeling Techniques to Enable Deep Generative Models in Engineering Designcited by 2
  14. 2025Human-in-the-Loop: Quantitative Evaluation of 3D Models Generation by Large Language Modelscited by 2
  15. 2025Are vision-language models ready to zero-shot replace supervised classification models in agriculture?cited by 2
  16. 2024Can Large Language Models Create New Knowledge for Spatial Reasoning Tasks?cited by 1
  17. 2024MSEval: A Dataset for Material Selection in Conceptual Design to Evaluate Algorithmic Modelscited by 1
  18. 2025Feedback-Driven Vision-Language Alignment with Minimal Human Supervisioncited by 1
  19. 2025Evaluation of Architectural Synthesis Using Generative AIcited by 1
  20. 2026Predictive Prototyping: Evaluating Design Concepts with ChatGPTcited by 1
  21. 2026Design-to-Plan: A Large Language Model-Based Multi-Agent Framework for Manufacturing Process Planning from 3D CAD Models and 2D Engineering Drawingscited by 1
  22. 2024DesignMinds: Enhancing Video-Based Design Ideation with Vision-Language Model and Context-Injected Large Language Modelcited by 0
  23. 2024CLAS: A Machine Learning Enhanced Framework for Exploring Large 3D Design Datasetscited by 0
  24. 2025Fine-Tuning Florence2 for Enhanced Object Detection in Un-constructed Environments: Vision-Language Model Approachcited by 0
  25. 2025AI Judges in Design: Statistical Perspectives on Achieving Human Expert Equivalence With Vision-Language Modelscited by 0
  26. 2025Masked Conditioning for Deep Generative Modelscited by 0
  27. 2025Intelligent Design 4.0: Paradigm Evolution Toward the Agentic AI Eracited by 0
  28. 2026Kolmogorov-Arnold Networks-Based Tolerance-Aware Manufacturability Assessment Integrating Design-for-Manufacturing Principlescited by 0
  29. 2026Context-Aware Mapping of 2D Drawing Annotations to 3D CAD Features Using LLM-Assisted Reasoning for Manufacturing Automationcited by 0
  30. 2026DUCTILE: Agentic LLM Orchestration of Engineering Analysis in Product Development Practicecited by 0
  31. 2026Agentic AI in Engineering and Manufacturing: Industry Perspectives on Utility, Adoption, Challenges, and Opportunitiescited by 0
  32. 2026Agentic Risk-Aware Set-Based Engineering Designcited by 0
  33. 2026Research on Vision-Language Question Answering Models for Industrial Robotscited by 0
  34. 2026CAX-Agent: A Lightweight Agent Harness for Reliable APDL Automationcited by 0
  35. 2026Surrogate Assisted Pedestrian Protection Design via a Foundation Model Orchestrated Workflowcited by 0
  36. 2026Enhancing Creativity in 3D Generative Design via a TRIZ-Inspired Text-to-CAD Frameworkcited by 0
  37. 2026PlanSightRAG: A Visual-First Multimodal RAG for Automating Question Answering and Compliance Checking for Civil Standard Planscited by 0
  38. 2026Learning to Ground Before Reading: Unified PCB Engineering Drawing Parsing with Compact Vision-Language Modelscited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024CadVLM: Bridging Language and Vision in the Generation of Parametric CAD Sketches
  2. 2024Leveraging Vision-Language Models for Manufacturing Feature Recognition in CAD Designs
  3. 2026Do VLMs Reason Like Engineers? A Benchmark and a Stage-wise Evaluation
  4. 2025A Multi-Stage Hybrid Framework for Automated Interpretation of Multi-View Engineering Drawings Using Vision Language Model
  5. 2026Enginuity: A Dataset and Benchmark for Vision-Language Understanding of Engineering Diagrams
  6. 2025CAD-Coder: An Open-Source Vision-Language Model for Computer-Aided Design Code Generation
  7. 2026Foundation Models for Automatic CAD Generation
  8. 2025From Drawings to Decisions: A Hybrid Vision-Language Framework for Parsing 2D Engineering Drawings into Structured Manufacturing Knowledge