# CAD-GPT: Synthesising CAD Construction Sequence with Spatial Reasoning-Enhanced Multimodal LLMs

arXiv: https://arxiv.org/abs/2412.19663 · HTML: https://mlsanity.com/paper/2412.19663/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 19 of its 31 references that are in the map.
- First posted on arXiv 27 Dec 2024.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 19 of 19)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation](https://mlsanity.com/paper/2309.16653/) (2023)
- [Diffusion Probabilistic Models for 3D Point Cloud Generation](https://mlsanity.com/paper/2103.01458/) (2021)
- [Point-E: A System for Generating 3D Point Clouds from Complex Prompts](https://mlsanity.com/paper/2212.08751/) (2022)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018)
- [3D Shape Generation and Completion through Point-Voxel Diffusion](https://mlsanity.com/paper/2104.03670/) (2021)
- [DeepCAD: A Deep Generative Network for Computer-Aided Design Models](https://mlsanity.com/paper/2105.09492/) (2021)
- [PolyGen: An Autoregressive Generative Model of 3D Meshes](https://mlsanity.com/paper/2002.10880/) (2020)
- [MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers](https://mlsanity.com/paper/2311.15475/) (2023)
- [GaussianEditor: Editing 3D Gaussians Delicately with Text Instructions](https://mlsanity.com/paper/2311.16037/) (2023)
- [MeshXL: Neural Coordinate Field for Generative 3D Foundation Models](https://mlsanity.com/paper/2405.20853/) (2024)
- [ParSeNet: A Parametric Surface Fitting Network for 3D Point Clouds](https://mlsanity.com/paper/2003.12181/) (2020)
- [Hierarchical Neural Coding for Controllable CAD Model Generation](https://mlsanity.com/paper/2307.00149/) (2023)
- [PIE-NET: Parametric Inference of Point Cloud Edges](https://mlsanity.com/paper/2007.04883/) (2020)
- [CAD-SIGNet: CAD Language Inference from Point Clouds using Layer-wise Sketch Instance Guided Attention](https://mlsanity.com/paper/2402.17678/) (2024)

## Built on by (most cited first, 30 of 53)

- [CAD-Llama: Leveraging Large Language Models for Computer-Aided Design Parametric 3D Model Generation](https://mlsanity.com/paper/2505.04481/) (2025)
- [LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?](https://mlsanity.com/paper/2503.19990/) (2025)
- [Seek-CAD: A Self-refined Generative Modeling for 3D Parametric CAD Using Local Inference via DeepSeek](https://mlsanity.com/paper/2505.17702/) (2025)
- [Large Language Models for Computer-Aided Design: A Survey](https://mlsanity.com/paper/2505.08137/) (2025)
- [cadrille: Multi-modal CAD Reconstruction with Reinforcement Learning](https://mlsanity.com/paper/2505.22914/) (2025)
- [BrepGPT: Autoregressive B-rep Generation with Voronoi Half-Patch](https://mlsanity.com/paper/2511.22171/) (2025)
- [ReCAD: Reinforcement Learning Enhanced Parametric CAD Model Generation with Vision-Language Models](https://mlsanity.com/paper/2512.06328/) (2025)
- [Pointer-CAD: Unifying B-Rep and Command Sequences via Pointer-based Edges & Faces Selection](https://mlsanity.com/paper/2603.04337/) (2026)
- [CADBench: A Multimodal Benchmark for AI-Assisted CAD Program Generation](https://mlsanity.com/paper/2605.10873/) (2026)
- [CAD-Judge: Toward Efficient Morphological Grading and Verification for Text-to-CAD Generation](https://mlsanity.com/paper/2508.04002/) (2025)
- [ArtiCAD: Articulated CAD Assembly Design via Multi-Agent Code Generation](https://mlsanity.com/paper/2604.10992/) (2026)
- [Zero-to-CAD: Agentic Synthesis of Interpretable CAD Programs at Million-Scale Without Real Data](https://mlsanity.com/paper/2604.24479/) (2026)
- [MUSE: Benchmarking Manufacturable, Functional, and Assemblable Text-to-CAD Generation](https://mlsanity.com/paper/2605.28579/) (2026)
- [Towards High-Fidelity CAD Generation via LLM-Driven Program Generation and Text-Based B-Rep Primitive Grounding](https://mlsanity.com/paper/2603.11831/) (2026)
- [CADReasoner: Iterative Program Editing for CAD Reverse Engineering](https://mlsanity.com/paper/2603.29847/) (2026)
- [TOOLCAD: Exploring Tool-Using Large Language Models in Text-to-CAD Generation with Reinforcement Learning](https://mlsanity.com/paper/2604.07960/) (2026)
- [GeoCAD: Local Geometry-Controllable CAD Generation with Large Language Models](https://mlsanity.com/paper/2506.10337/) (2025)
- [CADReview: Automatically Reviewing CAD Programs with Error Detection and Correction](https://mlsanity.com/paper/2505.22304/) (2025)
- [CADDesigner: Conceptual CAD Model Generation with a General-Purpose Agent](https://mlsanity.com/paper/2508.01031/) (2025)
- [CADFit: Precise Mesh-to-CAD Program Generation with Hybrid Optimization](https://mlsanity.com/paper/2605.01171/) (2026)
- [A Survey of AI Methods for Geometry Preparation and Mesh Generation in Engineering Simulation](https://mlsanity.com/paper/2512.23719/) (2025)
- [BrepCoder: A Unified Multimodal Large Language Model for Multi-task B-rep Reasoning](https://mlsanity.com/paper/2602.22284/) (2026)
- [PR-CAD: Progressive Refinement for Unified Controllable and Faithful Text-to-CAD Generation with Large Language Models](https://mlsanity.com/paper/2604.19773/) (2026)
- [Physics-in-the-Loop: A Hybrid Agentic Architecture for Validated CAD Engineering Design](https://mlsanity.com/paper/2605.19717/) (2026)
- [UniCAD: A Unified Benchmark and Universal Model for Multi-Modal Multi-Task CAD](https://mlsanity.com/paper/2606.05058/) (2026)
- [ArtisanCAD: An Industrial-Level CAD Agent with Expert-Grounded Knowledge Distillation](https://mlsanity.com/paper/2607.05750/) (2026)
- [GACO-CAD: Geometry-Augmented and Conciseness-Optimized CAD Model Generation from Single Image](https://mlsanity.com/paper/2510.17157/) (2025)
- [DreamCAD: Scaling Multi-modal CAD Generation using Differentiable Parametric Surfaces](https://mlsanity.com/paper/2603.05607/) (2026)
- [SldprtNet: A Large-Scale Multimodal Dataset for CAD Generation in Language-Driven 3D Design](https://mlsanity.com/paper/2603.13098/) (2026)
- [CAD-Tokenizer: Towards Text-based CAD Prototyping via Modality-Specific Tokenization](https://mlsanity.com/paper/2509.21150/) (2025)

## Abstract (arXiv)

Computer-aided design (CAD) significantly enhances the efficiency, accuracy, and innovation of design processes by enabling precise 2D and 3D modeling, extensive analysis, and optimization. Existing methods for creating CAD models rely on latent vectors or point clouds, which are difficult to obtain, and storage costs are substantial. Recent advances in Multimodal Large Language Models (MLLMs) have inspired researchers to use natural language instructions and images for CAD model construction. However, these models still struggle with inferring accurate 3D spatial location and orientation, leading to inaccuracies in determining the spatial 3D starting points and extrusion directions for constructing geometries. This work introduces CAD-GPT, a CAD synthesis method with spatial reasoning-enhanced MLLM that takes either a single image or a textual description as input. To achieve precise spatial inference, our approach introduces a 3D Modeling Spatial Mechanism. This method maps 3D spatial positions and 3D sketch plane rotation angles into a 1D linguistic feature space using a specialized spatial unfolding mechanism, while discretizing 2D sketch coordinates into an appropriate planar space to enable precise determination of spatial starting position, sketch orientation, and 2D sketch coordinate translations. Extensive experiments demonstrate that CAD-GPT consistently outperforms existing state-of-the-art methods in CAD model synthesis, both quantitatively and qualitatively.