mlsanityData to 14 Oct 2026

CAD-Assistant: Tool-Augmented VLLMs as Generic CAD Task Solvers

arXiv 2412.13810 ↗First posted 18 Dec 2024cs.CV cs.AI cs.LGcited by 40

Not on the map. Why?

Abstract (arXiv)

We propose CAD-Assistant, a general-purpose CAD agent for AI-assisted design. Our approach is based on a powerful Vision and Large Language Model (VLLM) as a planner and a tool-augmentation paradigm using CAD-specific tools. CAD-Assistant addresses multimodal user queries by generating actions that are iteratively executed on a Python interpreter equipped with the FreeCAD software, accessed via its Python API. Our framework is able to assess the impact of generated CAD commands on geometry and adapts subsequent actions based on the evolving state of the CAD design. We consider a wide range of CAD-specific tools including a sketch image parameterizer, rendering modules, a 2D cross-section generator, and other specialized routines. CAD-Assistant is evaluated on multiple CAD benchmarks, where it outperforms VLLM baselines and supervised task-specific methods. Beyond existing benchmarks, we qualitatively demonstrate the potential of tool-augmented VLLMs as general-purpose CAD solvers across diverse workflows.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 58 of its 78 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

58 of 78 references.

References · 58

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Segment Anythingcited by 8,234
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
38 more
  1. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  2. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  3. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  4. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  5. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  6. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  7. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  8. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  9. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  10. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  11. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 350
  12. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  13. 2023Large Language Models as Tool Makerscited by 303
  14. 2023Tool Learning with Foundation Modelscited by 302
  15. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
  16. 2021DeepCAD: A Deep Generative Network for Computer-Aided Design Modelscited by 271
  17. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 268
  18. 2021Internet-Augmented Dialogue Generationcited by 217
  19. 2022TALM: Tool Augmented Language Modelscited by 197
  20. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 159
  21. 2022SkexGen: Autoregressive Generation of CAD Construction Sequences with Disentangled Codebookscited by 125
  22. 2023Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Studycited by 116
  23. 2023Hierarchical Neural Coding for Controllable CAD Model Generationcited by 86
  24. 2020SketchGraphs: A Large-Scale Dataset for Modeling Relational Geometry in Computer-Aided Designcited by 72
  25. 2023Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Modelscited by 68
  26. 2021Vitruvion: A Generative Model of Parametric CAD Sketchescited by 66
  27. 2024CAD-SIGNet: CAD Language Inference from Point Clouds using Layer-wise Sketch Instance Guided Attentioncited by 64
  28. 2024Generating CAD Code with Vision-Language Models for 3D Designscited by 62
  29. 2021Point2Cyl: Reverse Engineering 3D Objects from Point Clouds to Extrusion Cylinderscited by 60
  30. 2020Deep Learning for Free-Hand Sketch: A Surveycited by 59
  31. 2023How Can Large Language Models Help Humans in Design and Manufacturing?cited by 41
  32. 2024A Vision Check-up for Language Modelscited by 36
  33. 2024Img2CAD: Reverse Engineering 3D CAD Models from Images through VLM-Assisted Conditional Factorizationcited by 35
  34. 2024Can Large Language Models Understand Symbolic Graphics Programs?cited by 35
  35. 2024Query2CAD: Generating CAD models using natural language queriescited by 29
  36. 2024TransCAD: A Hierarchical Transformer for CAD Sequence Inference from Point Cloudscited by 29
  37. 2024CadVLM: Bridging Language and Vision in the Generation of Parametric CAD Sketchescited by 28
  38. 2023NerVE: Neural Volumetric Edges for Parametric Curve Extraction from Point Cloudcited by 22

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 25.

40 papers.

Sort:

By area:

  1. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
  2. 2025Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learningcited by 49
  3. 2025CAD-Coder: An Open-Source Vision-Language Model for Computer-Aided Design Code Generationcited by 43
  4. 2025Seek-CAD: A Self-refined Generative Modeling for 3D Parametric CAD Using Local Inference via DeepSeekcited by 24
  5. 2025When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thoughtcited by 21
  6. 2025Large Language Models for Computer-Aided Design: A Surveycited by 18
  7. 2025cadrille: Multi-modal CAD Reconstruction with Reinforcement Learningcited by 16
  8. 2025VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoningcited by 12
  9. 2026Text-to-CAD Evaluation with CADTestscited by 10
  10. 2025BuildArena: A Physics-Aligned Interactive Benchmark of LLMs for Engineering Constructioncited by 9
  11. 2025CReFT-CAD: Boosting Orthographic Projection Reasoning for CAD via Reinforcement Fine-Tuningcited by 8
  12. 2026Towards High-Fidelity CAD Generation via LLM-Driven Program Generation and Text-Based B-Rep Primitive Groundingcited by 7
28 more
  1. 2026CADReasoner: Iterative Program Editing for CAD Reverse Engineeringcited by 6
  2. 2025MiCADangelo: Fine-Grained Reconstruction of Constrained CAD Models from 3D Scanscited by 5
  3. 2025CADDesigner: Conceptual CAD Model Generation with a General-Purpose Agentcited by 4
  4. 2026Embodied CAD: Solver-Grounded LLM Agents for Parametric B-Rep Assembly Modelingcited by 4
  5. 2025PHT-CAD: Efficient CAD Parametric Primitive Analysis with Progressive Hierarchical Tuningcited by 3
  6. 2026COSMO-Agent: Tool-Augmented Agent for Closed-loop Optimization,Simulation,and Modeling Orchestrationcited by 3
  7. 2026neuralCAD-Edit: An Expert Benchmark for Multimodal-Instructed 3D CAD Model Editingcited by 3
  8. 2026UniCAD: A Unified Benchmark and Universal Model for Multi-Modal Multi-Task CADcited by 3
  9. 2026ArtisanCAD: An Industrial-Level CAD Agent with Expert-Grounded Knowledge Distillationcited by 3
  10. 2026Self-Improving CAD Generation Agents with Finite Element Analysis as Feedbackcited by 2
  11. 2026CADIR: A Cross-Backend Editable Intermediate Representation for Agentic CAD Generationcited by 2
  12. 2025BrepLLM: Enabling Large Language Models to Understand Boundary Representationscited by 1
  13. 2026PLLM: Pseudo-Labeling Large Language Models for CAD Program Synthesiscited by 1
  14. 2026Pointer-CAD v2: Plan-Then-Construct CAD Generation with Dimension-Aware Parametric Precisioncited by 1
  15. 2026SOV-CAD: Stepwise Orthographic Views Guided CAD Modeling Sequence Reconstructioncited by 1
  16. 2026CADENA: Stepwise CAD Reverse Engineeringcited by 1
  17. 2025GAIR: GUI Automation via Information-Joint Reasoning and Group Reflectioncited by 0
  18. 2026TableVision: A Large-Scale Benchmark for Spatially Grounded Reasoning over Complex Hierarchical Tablescited by 0
  19. 2026Agent-Aided Design for Dynamic CAD Modelscited by 0
  20. 2026Memory-Augmented Reinforcement Learning Agent for CAD Generationcited by 0
  21. 2026Tool-Augmented Agent for Closed-loop Optimization,Simulation,and Modeling Orchestrationcited by 0
  22. 2026LLM-based Visual Code Completion for Aerospace Geometric Designcited by 0
  23. 2026Three-Phase Evaluation of AI-Assisted Software Development Life Cyclecited by 0
  24. 2026TraceCAD: Trace-Guided Repair for Agentic CAD Generationcited by 0
  25. 2026Test-Time Scaling for CAD Generation via Verifier-Free Consensus Selectioncited by 0
  26. 2026Reasoning with Image Generationcited by 0
  27. 2026PolyBridgeBench: Benchmarking Multimodal LLMs for Physics-Grounded Bridge Designcited by 0
  28. 2026Task-Adaptive Grounded 3D-Programmers Using 2D VLMscited by 0