# Human-in-the-Loop: Quantitative Evaluation of 3D Models Generation by Large Language Models

arXiv: https://arxiv.org/abs/2509.07010 · HTML: https://mlsanity.com/paper/2509.07010/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 38 references that are in the map.
- First posted on arXiv 6 Sep 2025.

## Builds on (most cited first, 16 of 16)

- [MechAgents: Large language model multi-agent collaborations can solve mechanics problems, generate new data, and integrate knowledge](https://mlsanity.com/paper/2311.08166/) (2023)
- [Generating CAD Code with Vision-Language Models for 3D Designs](https://mlsanity.com/paper/2410.05340/) (2024)
- [From Concept to Manufacturing: Evaluating Vision-Language Models for Engineering Design](https://mlsanity.com/paper/2311.12668/) (2023)
- [How Can Large Language Models Help Humans in Design and Manufacturing?](https://mlsanity.com/paper/2307.14377/) (2023)
- [DesignQA: A Multimodal Benchmark for Evaluating Large Language Models' Understanding of Engineering Documentation](https://mlsanity.com/paper/2404.07917/) (2024)
- [OpenECAD: An Efficient Visual Language Model for Editable 3D-CAD Design](https://mlsanity.com/paper/2406.09913/) (2024)
- [Generative Transformers for Design Concept Generation](https://mlsanity.com/paper/2211.03468/) (2022)
- [3D-PreMise: Can Large Language Models Generate 3D Shapes with Sharp Features and Parametric Control?](https://mlsanity.com/paper/2401.06437/) (2024)
- [Large Language Model Agent as a Mechanical Designer](https://mlsanity.com/paper/2404.17525/) (2024)
- [Generative Pre-Trained Transformer for Design Concept Generation: An Exploration](https://mlsanity.com/paper/2111.08489/) (2021)
- [Assessing Large Language Models in Mechanical Engineering Education: A Study on Mechanics-Focused Conceptual Understanding](https://mlsanity.com/paper/2401.12983/) (2024)
- [From Idea to CAD: A Language Model-Driven Multi-Agent System for Collaborative Design](https://mlsanity.com/paper/2503.04417/) (2025)
- [Biologically Inspired Design Concept Generation Using Generative Pre-Trained Transformers](https://mlsanity.com/paper/2212.13196/) (2022)
- [Analysis of ChatGPT on Source Code](https://mlsanity.com/paper/2306.00597/) (2023)
- [Large Language and Text-to-3D Models for Engineering Design Optimization](https://mlsanity.com/paper/2307.01230/) (2023)
- [Coding by Design: GPT-4 empowers Agile Model Driven Development](https://mlsanity.com/paper/2310.04304/) (2023)

## Built on by (most cited first, 2 of 2)

- [3D-CBM: A Framework for Concept-Based Interpretability in Generative 3D Modeling](https://mlsanity.com/paper/2606.11446/) (2026)
- [Toward AI-Augmented Cooperative Engineering Workflows: Requirements and Architecture the European Rover Challenge](https://mlsanity.com/paper/2609.31136/) (2026)

## Abstract (arXiv)

Large Language Models are increasingly capable of interpreting multimodal inputs to generate complex 3D shapes, yet robust methods to evaluate geometric and structural fidelity remain underdeveloped. This paper introduces a human in the loop framework for the quantitative evaluation of LLM generated 3D models, supporting applications such as democratization of CAD design, reverse engineering of legacy designs, and rapid prototyping. We propose a comprehensive suite of similarity and complexity metrics, including volumetric accuracy, surface alignment, dimensional fidelity, and topological intricacy, to benchmark generated models against ground truth CAD references. Using an L bracket component as a case study, we systematically compare LLM performance across four input modalities: 2D orthographic views, isometric sketches, geometric structure trees, and code based correction prompts. Our findings demonstrate improved generation fidelity with increased semantic richness, with code level prompts achieving perfect reconstruction across all metrics. A key contribution of this work is demonstrating that our proposed quantitative evaluation approach enables significantly faster convergence toward the ground truth, especially compared to traditional qualitative methods based solely on visual inspection and human intuition. This work not only advances the understanding of AI assisted shape synthesis but also provides a scalable methodology to validate and refine generative models for diverse CAD applications.