mlsanityData to 14 Oct 2026

CadVLM: Bridging Language and Vision in the Generation of Parametric CAD Sketches

arXiv 2409.17457 ↗First posted 26 Sep 2024cs.CV cs.AIcited by 28

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Parametric Computer-Aided Design (CAD) is central to contemporary mechanical design. However, it encounters challenges in achieving precise parametric sketch modeling and lacks practical evaluation metrics suitable for mechanical design. We harness the capabilities of pre-trained foundation models, renowned for their successes in natural language processing and computer vision, to develop generative models specifically for CAD. These models are adept at understanding complex geometries and design reasoning, a crucial advancement in CAD technology. In this paper, we propose CadVLM, an end-to-end vision language model for CAD generation. Our approach involves adapting pre-trained foundation models to manipulate engineering sketches effectively, integrating both sketch primitive sequences and sketch images. Extensive experiments demonstrate superior performance on multiple CAD sketch generation tasks such as CAD autocompletion, CAD autoconstraint, and image conditional generation. To our knowledge, this is the first instance of a multimodal Large Language Model (LLM) being successfully applied to parametric CAD generation, representing a pioneering step in the field of computer-aided mechanical design.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 33 references that are in the map. First posted on arXiv 26 Sep 2024.

Builds on

29 of 33 references.

References · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  10. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  11. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  12. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
17 more
  1. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  2. 2023Llemma: An Open Language Model For Mathematicscited by 388
  3. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 384
  4. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  5. 2021DeepCAD: A Deep Generative Network for Computer-Aided Design Modelscited by 271
  6. 2022SkexGen: Autoregressive Generation of CAD Construction Sequences with Disentangled Codebookscited by 125
  7. 2023Hierarchical Neural Coding for Controllable CAD Model Generationcited by 86
  8. 2021Computer-Aided Design as Languagecited by 82
  9. 2020SketchGraphs: A Large-Scale Dataset for Modeling Relational Geometry in Computer-Aided Designcited by 72
  10. 2021SketchGen: Generating Constrained CAD Sketchescited by 68
  11. 2021Vitruvion: A Generative Model of Parametric CAD Sketchescited by 66
  12. 2023TableGPT: Towards Unifying Tables, Nature Language and Commands into One GPTcited by 59
  13. 2023MedBLIP: Bootstrapping Language-Image Pre-training from 3D Medical Images and Textscited by 51
  14. 2021Engineering Sketch Generation for Computer-Aided Designcited by 49
  15. 2023SECAD-Net: Self-Supervised CAD Reconstruction by Learning Sketch-Extrude Operationscited by 46
  16. 2022Self-Supervised Representation Learning for CADcited by 22
  17. 2024CaMML: Context-Aware Multimodal Learner for Large Modelscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 13; 2026: 11.

28 papers.

Sort:

By area:

  1. 2024CAD-MLLM: Unifying Multimodality-Conditioned CAD Generation With MLLMcited by 81
  2. 2025CAD-Coder: An Open-Source Vision-Language Model for Computer-Aided Design Code Generationcited by 43
  3. 2024CAD-Assistant: Tool-Augmented VLLMs as Generic CAD Task Solverscited by 40
  4. 2024BlenderLLM: Training Large Language Models for Computer-Aided Design with Self-improvementcited by 29
  5. 2025Large Language Models for Computer-Aided Design: A Surveycited by 18
  6. 2024Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptionscited by 13
  7. 2025ReCAD: Reinforcement Learning Enhanced Parametric CAD Model Generation with Vision-Language Modelscited by 11
  8. 2025Aligning Constraint Generation with Design Intent in Parametric CADcited by 8
  9. 2025Automated CAD Modeling Sequence Generation from Text Descriptions via Transformer-Based Large Language Modelscited by 8
  10. 2025GeoCAD: Local Geometry-Controllable CAD Generation with Large Language Modelscited by 5
  11. 2025MiCADangelo: Fine-Grained Reconstruction of Constrained CAD Models from 3D Scanscited by 5
  12. 2025CADReview: Automatically Reviewing CAD Programs with Error Detection and Correctioncited by 4
16 more
  1. 2025CADDesigner: Conceptual CAD Model Generation with a General-Purpose Agentcited by 4
  2. 2025PHT-CAD: Efficient CAD Parametric Primitive Analysis with Progressive Hierarchical Tuningcited by 3
  3. 2025A Survey of AI Methods for Geometry Preparation and Mesh Generation in Engineering Simulationcited by 3
  4. 2026neuralCAD-Edit: An Expert Benchmark for Multimodal-Instructed 3D CAD Model Editingcited by 3
  5. 2025BrepLLM: Enabling Large Language Models to Understand Boundary Representationscited by 1
  6. 2026STEP-LLM: Generating CAD STEP Models from Natural Language with Large Language Modelscited by 1
  7. 2026Pointer-CAD v2: Plan-Then-Construct CAD Generation with Dimension-Aware Parametric Precisioncited by 1
  8. 2026CADENA: Stepwise CAD Reverse Engineeringcited by 1
  9. 2025Temporal-Visual Semantic Alignment: A Unified Architecture for Transferring Spatial Priors from Vision Models to Zero-Shot Temporal Taskscited by 0
  10. 2026Flatten The Complex: Joint B-Rep Generation via Compositional k-Cell Particlescited by 0
  11. 2026Computer-Aided Design Generation by Cascaded Discrete Diffusion Modelcited by 0
  12. 2026Memory-Augmented Reinforcement Learning Agent for CAD Generationcited by 0
  13. 2026GuideCAD: A Lightweight Multimodal Framework for 3D CAD Model Generation via Prefix Embeddingcited by 0
  14. 2026LLM-based Visual Code Completion for Aerospace Geometric Designcited by 0
  15. 2026VisCAD: A Foundation Model Suite with Multimodal Industrial CAD Intelligencecited by 0
  16. 2026AgenticCADedit: A Stateful, Tool-Mediated Agentic Approach to Multimodal 3D CAD Editingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CADFS: A Big CAD Program Dataset and Framework for Computer-Aided Design with Large Language Models
  2. 2026Foundation Models for Automatic CAD Generation
  3. 2023From Concept to Manufacturing: Evaluating Vision-Language Models for Engineering Design
  4. 2024CAD-GPT: Synthesising CAD Construction Sequence with Spatial Reasoning-Enhanced Multimodal LLMs
  5. 2024From 2D CAD Drawings to 3D Parametric Models: A Vision-Language Approach
  6. 2025Seek-CAD: A Self-refined Generative Modeling for 3D Parametric CAD Using Local Inference via DeepSeek
  7. 2024Generating CAD Code with Vision-Language Models for 3D Designs
  8. 2025A Solver-Aided Hierarchical Language for LLM-Driven CAD Design