# CadVLM: Bridging Language and Vision in the Generation of Parametric CAD Sketches

arXiv: https://arxiv.org/abs/2409.17457 · HTML: https://mlsanity.com/paper/2409.17457/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 29 of its 33 references that are in the map.
- First posted on arXiv 26 Sep 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct](https://mlsanity.com/paper/2308.09583/) (2023)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Llemma: An Open Language Model For Mathematics](https://mlsanity.com/paper/2310.10631/) (2023)
- [Towards Expert-Level Medical Question Answering with Large Language Models](https://mlsanity.com/paper/2305.09617/) (2023)
- [CodeT5+: Open Code Large Language Models for Code Understanding and Generation](https://mlsanity.com/paper/2305.07922/) (2023)
- [DeepCAD: A Deep Generative Network for Computer-Aided Design Models](https://mlsanity.com/paper/2105.09492/) (2021)
- [SkexGen: Autoregressive Generation of CAD Construction Sequences with Disentangled Codebooks](https://mlsanity.com/paper/2207.04632/) (2022)
- [Hierarchical Neural Coding for Controllable CAD Model Generation](https://mlsanity.com/paper/2307.00149/) (2023)
- [Computer-Aided Design as Language](https://mlsanity.com/paper/2105.02769/) (2021)
- [SketchGraphs: A Large-Scale Dataset for Modeling Relational Geometry in Computer-Aided Design](https://mlsanity.com/paper/2007.08506/) (2020)
- [SketchGen: Generating Constrained CAD Sketches](https://mlsanity.com/paper/2106.02711/) (2021)
- [Vitruvion: A Generative Model of Parametric CAD Sketches](https://mlsanity.com/paper/2109.14124/) (2021)
- [TableGPT: Towards Unifying Tables, Nature Language and Commands into One GPT](https://mlsanity.com/paper/2307.08674/) (2023)
- [MedBLIP: Bootstrapping Language-Image Pre-training from 3D Medical Images and Texts](https://mlsanity.com/paper/2305.10799/) (2023)
- [Engineering Sketch Generation for Computer-Aided Design](https://mlsanity.com/paper/2104.09621/) (2021)
- [SECAD-Net: Self-Supervised CAD Reconstruction by Learning Sketch-Extrude Operations](https://mlsanity.com/paper/2303.10613/) (2023)
- [Self-Supervised Representation Learning for CAD](https://mlsanity.com/paper/2210.10807/) (2022)
- [CaMML: Context-Aware Multimodal Learner for Large Models](https://mlsanity.com/paper/2401.03149/) (2024)

## Built on by (most cited first, 28 of 28)

- [CAD-MLLM: Unifying Multimodality-Conditioned CAD Generation With MLLM](https://mlsanity.com/paper/2411.04954/) (2024)
- [CAD-Coder: An Open-Source Vision-Language Model for Computer-Aided Design Code Generation](https://mlsanity.com/paper/2505.14646/) (2025)
- [CAD-Assistant: Tool-Augmented VLLMs as Generic CAD Task Solvers](https://mlsanity.com/paper/2412.13810/) (2024)
- [BlenderLLM: Training Large Language Models for Computer-Aided Design with Self-improvement](https://mlsanity.com/paper/2412.14203/) (2024)
- [Large Language Models for Computer-Aided Design: A Survey](https://mlsanity.com/paper/2505.08137/) (2025)
- [Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptions](https://mlsanity.com/paper/2412.08737/) (2024)
- [ReCAD: Reinforcement Learning Enhanced Parametric CAD Model Generation with Vision-Language Models](https://mlsanity.com/paper/2512.06328/) (2025)
- [Aligning Constraint Generation with Design Intent in Parametric CAD](https://mlsanity.com/paper/2504.13178/) (2025)
- [Automated CAD Modeling Sequence Generation from Text Descriptions via Transformer-Based Large Language Models](https://mlsanity.com/paper/2505.19490/) (2025)
- [GeoCAD: Local Geometry-Controllable CAD Generation with Large Language Models](https://mlsanity.com/paper/2506.10337/) (2025)
- [MiCADangelo: Fine-Grained Reconstruction of Constrained CAD Models from 3D Scans](https://mlsanity.com/paper/2510.23429/) (2025)
- [CADReview: Automatically Reviewing CAD Programs with Error Detection and Correction](https://mlsanity.com/paper/2505.22304/) (2025)
- [CADDesigner: Conceptual CAD Model Generation with a General-Purpose Agent](https://mlsanity.com/paper/2508.01031/) (2025)
- [PHT-CAD: Efficient CAD Parametric Primitive Analysis with Progressive Hierarchical Tuning](https://mlsanity.com/paper/2503.18147/) (2025)
- [A Survey of AI Methods for Geometry Preparation and Mesh Generation in Engineering Simulation](https://mlsanity.com/paper/2512.23719/) (2025)
- [neuralCAD-Edit: An Expert Benchmark for Multimodal-Instructed 3D CAD Model Editing](https://mlsanity.com/paper/2604.16170/) (2026)
- [BrepLLM: Enabling Large Language Models to Understand Boundary Representations](https://mlsanity.com/paper/2512.16413/) (2025)
- [STEP-LLM: Generating CAD STEP Models from Natural Language with Large Language Models](https://mlsanity.com/paper/2601.12641/) (2026)
- [Pointer-CAD v2: Plan-Then-Construct CAD Generation with Dimension-Aware Parametric Precision](https://mlsanity.com/paper/2606.29301/) (2026)
- [CADENA: Stepwise CAD Reverse Engineering](https://mlsanity.com/paper/2608.00799/) (2026)
- [Temporal-Visual Semantic Alignment: A Unified Architecture for Transferring Spatial Priors from Vision Models to Zero-Shot Temporal Tasks](https://mlsanity.com/paper/2511.19856/) (2025)
- [Flatten The Complex: Joint B-Rep Generation via Compositional k-Cell Particles](https://mlsanity.com/paper/2601.17733/) (2026)
- [Computer-Aided Design Generation by Cascaded Discrete Diffusion Model](https://mlsanity.com/paper/2605.05031/) (2026)
- [Memory-Augmented Reinforcement Learning Agent for CAD Generation](https://mlsanity.com/paper/2605.19748/) (2026)
- [GuideCAD: A Lightweight Multimodal Framework for 3D CAD Model Generation via Prefix Embedding](https://mlsanity.com/paper/2606.07024/) (2026)
- [LLM-based Visual Code Completion for Aerospace Geometric Design](https://mlsanity.com/paper/2606.16806/) (2026)
- [VisCAD: A Foundation Model Suite with Multimodal Industrial CAD Intelligence](https://mlsanity.com/paper/2609.03811/) (2026)
- [AgenticCADedit: A Stateful, Tool-Mediated Agentic Approach to Multimodal 3D CAD Editing](https://mlsanity.com/paper/2609.29621/) (2026)

## Abstract (arXiv)

Parametric Computer-Aided Design (CAD) is central to contemporary mechanical design. However, it encounters challenges in achieving precise parametric sketch modeling and lacks practical evaluation metrics suitable for mechanical design. We harness the capabilities of pre-trained foundation models, renowned for their successes in natural language processing and computer vision, to develop generative models specifically for CAD. These models are adept at understanding complex geometries and design reasoning, a crucial advancement in CAD technology. In this paper, we propose CadVLM, an end-to-end vision language model for CAD generation. Our approach involves adapting pre-trained foundation models to manipulate engineering sketches effectively, integrating both sketch primitive sequences and sketch images. Extensive experiments demonstrate superior performance on multiple CAD sketch generation tasks such as CAD autocompletion, CAD autoconstraint, and image conditional generation. To our knowledge, this is the first instance of a multimodal Large Language Model (LLM) being successfully applied to parametric CAD generation, representing a pioneering step in the field of computer-aided mechanical design.