# From Concept to Manufacturing: Evaluating Vision-Language Models for Engineering Design

arXiv: https://arxiv.org/abs/2311.12668 · HTML: https://mlsanity.com/paper/2311.12668/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 16 of its 103 references that are in the map.
- First posted on arXiv 21 Nov 2023.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 16 of 16)

- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V](https://mlsanity.com/paper/2310.11441/) (2023)
- [GPTEval: A Survey on Assessments of ChatGPT and GPT-4](https://mlsanity.com/paper/2308.12488/) (2023)
- [TopologyGAN: Topology Optimization Using Generative Adversarial Networks Based on Physical Fields Over the Initial Domain](https://mlsanity.com/paper/2003.04685/) (2020)
- [Aligning Optimization Trajectories with Diffusion Models for Constrained Design Generation](https://mlsanity.com/paper/2305.18470/) (2023)
- [On the use of Artificial Neural Networks in Topology Optimisation](https://mlsanity.com/paper/2208.02563/) (2022)
- [MeLM, a generative pretrained language modeling framework that solves forward and inverse mechanics problems](https://mlsanity.com/paper/2306.17525/) (2023)
- [Multi-modal Machine Learning in Engineering Design: A Review and Future Directions](https://mlsanity.com/paper/2302.10909/) (2023)
- [Meta-learning Convolutional Neural Architectures for Multi-target Concrete Defect Classification with the COncrete DEfect BRidge IMage Dataset](https://mlsanity.com/paper/1904.08486/) (2019)
- [GPT Models in Construction Industry: Opportunities, Limitations, and a Use Case Validation](https://mlsanity.com/paper/2305.18997/) (2023)
- [Natural Language Processing in-and-for Design Research](https://mlsanity.com/paper/2111.13827/) (2021)
- [Multi-modal Machine Learning for Vehicle Rating Predictions Using Image, Text, and Parametric Data](https://mlsanity.com/paper/2305.15218/) (2023)

## Built on by (most cited first, 30 of 51)

- [DesignQA: A Multimodal Benchmark for Evaluating Large Language Models' Understanding of Engineering Documentation](https://mlsanity.com/paper/2404.07917/) (2024)
- [Large Language Models for Manufacturing](https://mlsanity.com/paper/2410.21418/) (2024)
- [Large Language Model Agent as a Mechanical Designer](https://mlsanity.com/paper/2404.17525/) (2024)
- [Literature-Grounded Novelty Assessment of Scientific Ideas](https://mlsanity.com/paper/2506.22026/) (2025)
- [LLM-3D Print: Large Language Models To Monitor and Control 3D Printing](https://mlsanity.com/paper/2408.14307/) (2024)
- [Large Language Models for Computer-Aided Design: A Survey](https://mlsanity.com/paper/2505.08137/) (2025)
- [From Idea to CAD: A Language Model-Driven Multi-Agent System for Collaborative Design](https://mlsanity.com/paper/2503.04417/) (2025)
- [Sketch2Prototype: Rapid Conceptual Design Exploration and Prototyping with Generative AI](https://mlsanity.com/paper/2405.12985/) (2024)
- [Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptions](https://mlsanity.com/paper/2412.08737/) (2024)
- [CADBench: A Multimodal Benchmark for AI-Assisted CAD Program Generation](https://mlsanity.com/paper/2605.10873/) (2026)
- [Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideas](https://mlsanity.com/paper/2603.10303/) (2026)
- [AutoTRIZ: Automating Engineering Innovation with TRIZ and Large Language Models](https://mlsanity.com/paper/2403.13002/) (2024)
- [SoM-1K: A Thousand-Problem Benchmark Dataset for Strength of Materials](https://mlsanity.com/paper/2509.21079/) (2025)
- [Agentic Large Language Models for Conceptual Systems Engineering and Design](https://mlsanity.com/paper/2507.08619/) (2025)
- [Distilled Large Language Model-Driven Dynamic Sparse Expert Activation Mechanism](https://mlsanity.com/paper/2603.26735/) (2026)
- [EvoCAD: Evolutionary CAD Code Generation with Vision Language Models](https://mlsanity.com/paper/2510.11631/) (2025)
- [From Cloud to Edge: Rethinking Generative AI for Low-Resource Design Challenges](https://mlsanity.com/paper/2402.12702/) (2024)
- [Toward Autonomous Engineering Design: A Knowledge-Guided Multi-Agent Framework](https://mlsanity.com/paper/2511.03179/) (2025)
- [Industrial Language-Image Dataset (ILID): Adapting Vision Foundation Models for Industrial Settings](https://mlsanity.com/paper/2406.09637/) (2024)
- [MechVQA: Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understanding](https://mlsanity.com/paper/2605.30794/) (2026)
- [Exploring the Potentials and Challenges of Deep Generative Models in Product Design Conception](https://mlsanity.com/paper/2407.11104/) (2024)
- [Towards an automated workflow in materials science for combining multi-modal simulative and experimental information using data mining and large language models](https://mlsanity.com/paper/2502.14904/) (2025)
- [GRAID: Enhancing Spatial Reasoning of VLMs Through High-Fidelity Data Generation](https://mlsanity.com/paper/2510.22118/) (2025)
- [Evaluating Large Language Models for Material Selection](https://mlsanity.com/paper/2405.03695/) (2024)
- [GeoBiked: A Dataset with Geometric Features and Automated Labeling Techniques to Enable Deep Generative Models in Engineering Design](https://mlsanity.com/paper/2409.17045/) (2024)
- [Human-in-the-Loop: Quantitative Evaluation of 3D Models Generation by Large Language Models](https://mlsanity.com/paper/2509.07010/) (2025)
- [Are vision-language models ready to zero-shot replace supervised classification models in agriculture?](https://mlsanity.com/paper/2512.15977/) (2025)
- [Can Large Language Models Create New Knowledge for Spatial Reasoning Tasks?](https://mlsanity.com/paper/2405.14379/) (2024)
- [MSEval: A Dataset for Material Selection in Conceptual Design to Evaluate Algorithmic Models](https://mlsanity.com/paper/2407.09719/) (2024)
- [Feedback-Driven Vision-Language Alignment with Minimal Human Supervision](https://mlsanity.com/paper/2501.04568/) (2025)

## Abstract (arXiv)

Engineering design is undergoing a transformative shift with the advent of AI, marking a new era in how we approach product, system, and service planning. Large language models have demonstrated impressive capabilities in enabling this shift. Yet, with text as their only input modality, they cannot leverage the large body of visual artifacts that engineers have used for centuries and are accustomed to. This gap is addressed with the release of multimodal vision-language models (VLMs), such as GPT-4V, enabling AI to impact many more types of tasks. Our work presents a comprehensive evaluation of VLMs across a spectrum of engineering design tasks, categorized into four main areas: Conceptual Design, System-Level and Detailed Design, Manufacturing and Inspection, and Engineering Education Tasks. Specifically in this paper, we assess the capabilities of two VLMs, GPT-4V and LLaVA 1.6 34B, in design tasks such as sketch similarity analysis, CAD generation, topology optimization, manufacturability assessment, and engineering textbook problems. Through this structured evaluation, we not only explore VLMs' proficiency in handling complex design challenges but also identify their limitations in complex engineering design applications. Our research establishes a foundation for future assessments of vision language models. It also contributes a set of benchmark testing datasets, with more than 1000 queries, for ongoing advancements and applications in this field.