# Text-to-CAD Generation Through Infusing Visual Feedback in Large Language Models

arXiv: https://arxiv.org/abs/2501.19054 · HTML: https://mlsanity.com/paper/2501.19054/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 31 Jan 2025.

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback](https://mlsanity.com/paper/2309.00267/) (2023)
- [DeepCAD: A Deep Generative Network for Computer-Aided Design Models](https://mlsanity.com/paper/2105.09492/) (2021)
- [Rich Human Feedback for Text-to-Image Generation](https://mlsanity.com/paper/2312.10240/) (2023)
- [SkexGen: Autoregressive Generation of CAD Construction Sequences with Disentangled Codebooks](https://mlsanity.com/paper/2207.04632/) (2022)
- [Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer](https://mlsanity.com/paper/2405.16436/) (2024)
- [BrepGen: A B-rep Generative Diffusion Model with Structured Latent Geometry](https://mlsanity.com/paper/2401.15563/) (2024)
- [Hierarchical Neural Coding for Controllable CAD Model Generation](https://mlsanity.com/paper/2307.00149/) (2023)
- [SolidGen: An Autoregressive Model for Direct B-rep Synthesis](https://mlsanity.com/paper/2203.13944/) (2022)
- [UCSG-Net -- Unsupervised Discovering of Constructive Solid Geometry Tree](https://mlsanity.com/paper/2006.09102/) (2020)
- [Vitruvion: A Generative Model of Parametric CAD Sketches](https://mlsanity.com/paper/2109.14124/) (2021)
- [CAD-SIGNet: CAD Language Inference from Point Clouds using Layer-wise Sketch Instance Guided Attention](https://mlsanity.com/paper/2402.17678/) (2024)
- [CAPRI-Net: Learning Compact CAD Shapes with Adaptive Primitive Assembly](https://mlsanity.com/paper/2104.05652/) (2021)
- [Engineering Sketch Generation for Computer-Aided Design](https://mlsanity.com/paper/2104.09621/) (2021)
- [How Can Large Language Models Help Humans in Design and Manufacturing?](https://mlsanity.com/paper/2307.14377/) (2023)
- [D2CSG: Unsupervised Learning of Compact CSG Trees with Dual Complements and Dropouts](https://mlsanity.com/paper/2301.11497/) (2023)
- [Knowledge-to-SQL: Enhancing SQL Generation with Data Expert LLM](https://mlsanity.com/paper/2402.11517/) (2024)
- [LongReward: Improving Long-context Large Language Models with AI Feedback](https://mlsanity.com/paper/2410.21252/) (2024)
- [FlexCAD: Unified and Versatile Controllable CAD Generation with Fine-tuned Large Language Models](https://mlsanity.com/paper/2411.05823/) (2024)
- [Text2CAD: Generating Sequential CAD Models from Beginner-to-Expert Level Text Prompts](https://mlsanity.com/paper/2409.17106/) (2024)
- [Neural Face Identification in a 2D Wireframe Projection of a Manifold Object](https://mlsanity.com/paper/2203.04229/) (2022)

## Built on by (most cited first, 30 of 47)

- [CADmium: Fine-Tuning Code Language Models for Text-Driven Sequential CAD Design](https://mlsanity.com/paper/2507.09792/) (2025)
- [Seek-CAD: A Self-refined Generative Modeling for 3D Parametric CAD Using Local Inference via DeepSeek](https://mlsanity.com/paper/2505.17702/) (2025)
- [cadrille: Multi-modal CAD Reconstruction with Reinforcement Learning](https://mlsanity.com/paper/2505.22914/) (2025)
- [Text2CAD-Bench: A Benchmark for LLM-based Text-to-Parametric CAD Generation](https://mlsanity.com/paper/2605.18430/) (2026)
- [UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter-Efficient Fine-Tuning of Large Models](https://mlsanity.com/paper/2505.20154/) (2025)
- [Pointer-CAD: Unifying B-Rep and Command Sequences via Pointer-based Edges & Faces Selection](https://mlsanity.com/paper/2603.04337/) (2026)
- [CAD-Judge: Toward Efficient Morphological Grading and Verification for Text-to-CAD Generation](https://mlsanity.com/paper/2508.04002/) (2025)
- [Text-to-CAD Evaluation with CADTests](https://mlsanity.com/paper/2605.07807/) (2026)
- [Automated CAD Modeling Sequence Generation from Text Descriptions via Transformer-Based Large Language Models](https://mlsanity.com/paper/2505.19490/) (2025)
- [MUSE: Benchmarking Manufacturable, Functional, and Assemblable Text-to-CAD Generation](https://mlsanity.com/paper/2605.28579/) (2026)
- [Clarify Before You Draw: Proactive Agents for Robust Text-to-CAD Generation](https://mlsanity.com/paper/2602.03045/) (2026)
- [Towards High-Fidelity CAD Generation via LLM-Driven Program Generation and Text-Based B-Rep Primitive Grounding](https://mlsanity.com/paper/2603.11831/) (2026)
- [Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence](https://mlsanity.com/paper/2606.15932/) (2026)
- [GeoCAD: Local Geometry-Controllable CAD Generation with Large Language Models](https://mlsanity.com/paper/2506.10337/) (2025)
- [CADFS: A Big CAD Program Dataset and Framework for Computer-Aided Design with Large Language Models](https://mlsanity.com/paper/2605.01925/) (2026)
- [GIFT: Bootstrapping Image-to-CAD Program Synthesis via Geometric Feedback](https://mlsanity.com/paper/2603.27448/) (2026)
- [neuralCAD-Edit: An Expert Benchmark for Multimodal-Instructed 3D CAD Model Editing](https://mlsanity.com/paper/2604.16170/) (2026)
- [Physics-in-the-Loop: A Hybrid Agentic Architecture for Validated CAD Engineering Design](https://mlsanity.com/paper/2605.19717/) (2026)
- [ArtisanCAD: An Industrial-Level CAD Agent with Expert-Grounded Knowledge Distillation](https://mlsanity.com/paper/2607.05750/) (2026)
- [Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents](https://mlsanity.com/paper/2510.17491/) (2025)
- [CADKnitter: Compositional CAD Generation from Text and Geometry Guidance](https://mlsanity.com/paper/2512.11199/) (2025)
- [DreamCAD: Scaling Multi-modal CAD Generation using Differentiable Parametric Surfaces](https://mlsanity.com/paper/2603.05607/) (2026)
- [Nova3D: Code-Native Generation of Programmable 3D Assets](https://mlsanity.com/paper/2607.22738/) (2026)
- [CAD-Tokenizer: Towards Text-based CAD Prototyping via Modality-Specific Tokenization](https://mlsanity.com/paper/2509.21150/) (2025)
- [PLLM: Pseudo-Labeling Large Language Models for CAD Program Synthesis](https://mlsanity.com/paper/2602.12561/) (2026)
- [BRepCLIP: Contrastive Multimodal Pretraining on BRep Primitives for CAD Understanding](https://mlsanity.com/paper/2606.05515/) (2026)
- [FllumaOne: A Code-Native Multimodal CAD Dataset with Executable Programs and Kernel-Validated Feature Histories](https://mlsanity.com/paper/2606.17696/) (2026)
- [BIM-Edit: Benchmarking Large Language Models for IFC-Based Building Information Modeling](https://mlsanity.com/paper/2606.20146/) (2026)
- [SOV-CAD: Stepwise Orthographic Views Guided CAD Modeling Sequence Reconstruction](https://mlsanity.com/paper/2607.04119/) (2026)
- [HierCAD: Hierarchical Text-to-CAD Design via Structure Alignment and Parameter Grounding](https://mlsanity.com/paper/2607.11339/) (2026)

## Abstract (arXiv)

Creating Computer-Aided Design (CAD) models requires significant expertise and effort. Text-to-CAD, which converts textual descriptions into CAD parametric sequences, is crucial in streamlining this process. Recent studies have utilized ground-truth parametric sequences, known as sequential signals, as supervision to achieve this goal. However, CAD models are inherently multimodal, comprising parametric sequences and corresponding rendered visual objects. Besides,the rendering process from parametric sequences to visual objects is many-to-one. Therefore, both sequential and visual signals are critical for effective training. In this work, we introduce CADFusion, a framework that uses Large Language Models (LLMs) as the backbone and alternates between two training stages: the sequential learning (SL) stage and the visual feedback (VF) stage. In the SL stage, we train LLMs using ground-truth parametric sequences, enabling the generation of logically coherent parametric sequences. In the VF stage, we reward parametric sequences that render into visually preferred objects and penalize those that do not, allowing LLMs to learn how rendered visual objects are perceived and evaluated. These two stages alternate throughout the training, ensuring balanced learning and preserving benefits of both signals. Experiments demonstrate that CADFusion significantly improves performance, both qualitatively and quantitatively.