mlsanityData to 7 Oct 2026

3D-GPT: Procedural 3D Modeling with Large Language Models

arXiv 2310.12945 ↗First posted 19 Oct 2023cs.CV cs.GR cs.LGcited by 91

Not on the map. Why?

Abstract (arXiv)

In the pursuit of efficient automated content creation, procedural generation, leveraging modifiable parameters and rule-based systems, emerges as a promising approach. Nonetheless, it could be a demanding endeavor, given its intricate nature necessitating a deep understanding of rules, algorithms, and parameters. To reduce workload, we introduce 3D-GPT, a framework utilizing large language models~(LLMs) for instruction-driven 3D modeling. 3D-GPT positions LLMs as proficient problem solvers, dissecting the procedural 3D modeling tasks into accessible segments and appointing the apt agent for each task. 3D-GPT integrates three core agents: the task dispatch agent, the conceptualization agent, and the modeling agent. They collaboratively achieve two objectives. First, it enhances concise initial scene descriptions, evolving them into detailed forms while dynamically adapting the text based on subsequent instructions. Second, it integrates procedural generation, extracting parameter values from enriched text to effortlessly interface with 3D software for asset creation. Our empirical investigations confirm that 3D-GPT not only interprets and executes instructions, delivering reliable results but also collaborates effectively with human designers. Furthermore, it seamlessly integrates with Blender, unlocking expanded manipulation possibilities. Our work highlights the potential of LLMs in 3D modeling, offering a basic framework for future advancements in scene generation and animation.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Builds on 38 of its 46 references that are in the map. First posted on arXiv 19 Oct 2023.

Builds on

38 of 46 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  11. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  12. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
26 more
  1. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  2. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  3. 2022Galactica: A Large Language Model for Sciencecited by 736
  4. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  5. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 555
  6. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  7. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  8. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  9. 2022Kubric: A scalable dataset generatorcited by 373
  10. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  11. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  12. 2022Visual Classification via Description from Large Language Modelscited by 331
  13. 2023RealFusion: 360deg Reconstruction of Any Object from a Single Imagecited by 305
  14. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  15. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  16. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  17. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  18. 2023DreamBooth3D: Subject-Driven Text-to-3D Generationcited by 235
  19. 2023Improving CLIP Training with Language Rewritescited by 216
  20. 2022Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Modelscited by 151
  21. 2023Infinite Photorealistic Worlds using Procedural Generationcited by 142
  22. 2022ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reportscited by 100
  23. 2023ProAgent: Building Proactive Cooperative Agents with Large Language Modelscited by 97
  24. 2023MindAgent: Emergent Gaming Interactioncited by 87
  25. 2017Configurable 3D Scene Synthesis and 2D Image Rendering with Per-Pixel Ground Truth using Stochastic Grammarscited by 25
  26. 2021Semi-synthesis: A fast way to produce effective datasets for stereo matchingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 27; 2025: 28; 2026: 35.

91 papers.

Sort:

By area:

  1. 2023XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchiescited by 183
  2. 2024SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Codecited by 125
  3. 2024GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splattingcited by 104
  4. 2024VFusion3D: Learning Scalable 3D Generative Models from Video Diffusion Modelscited by 56
  5. 2024SceneTeller: Language-to-3D Scene Generationcited by 43
  6. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  7. 2024MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D Worldcited by 38
  8. 2024Chat2SVG: Vector Graphics Generation with Large Language Models and Image Diffusion Modelscited by 36
  9. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  10. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  11. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  12. 2024The Scene Language: Representing Scenes with Programs, Words, and Embeddingscited by 31
38 more
  1. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 31
  2. 2025BlenderGym: Benchmarking Foundational Model Systems for Graphics Editingcited by 27
  3. 2024CityCraft: A Real Crafter for 3D City Generationcited by 25
  4. 20253D Scene Generation: A Surveycited by 24
  5. 2024ChatGarment: Garment Estimation, Generation and Editing via Large Language Modelscited by 22
  6. 20243D-PreMise: Can Large Language Models Generate 3D Shapes with Sharp Features and Parametric Control?cited by 21
  7. 2024Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from User's Casual Sketchescited by 21
  8. 2024Re-Thinking Inverse Graphics With Large Language Modelscited by 20
  9. 2024CityX: Controllable Procedural Content Generation for Unbounded 3D Citiescited by 20
  10. 2024Chat-Edit-3D: Interactive 3D Scene Editing via Text Promptscited by 19
  11. 2024Trans4D: Realistic Geometry-Aware Transition for Compositional Text-to-4D Synthesiscited by 19
  12. 2024BlenderAlchemy: Editing 3D Graphics with Vision-Language Modelscited by 18
  13. 2025LL3M: Large Language 3D Modelerscited by 18
  14. 2025IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Renderingcited by 17
  15. 2025HOLODECK 2.0: Vision-Language-Guided 3D World Generation with Editingcited by 14
  16. 2025SPATIALGEN: Layout-guided 3D Indoor Scene Generationcited by 14
  17. 2024Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curationcited by 12
  18. 2026Code2Worlds: Empowering Coding LLMs for 4D World Generationcited by 12
  19. 2024Text2BIM: Generating Building Models Using a Large Language Model-based Multi-Agent Frameworkcited by 9
  20. 2025VLMaterial: Procedural Material Generation with Large Vision-Language Modelscited by 9
  21. 2025TACO: Think-Answer Consistency for Optimized Long-Chain Reasoning and Efficient Data Learning via Reinforcement Learning in LVLMscited by 9
  22. 20263DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Codecited by 9
  23. 2025Scaling Sequence-to-Sequence Generative Neural Renderingcited by 8
  24. 2024Reconstructing Animals and the Wildcited by 7
  25. 2024DI-PCG: Diffusion-based Efficient Inverse Procedural Content Generation for High-quality 3D Asset Creationcited by 7
  26. 2026SceneCode: Executable World Programs for Editable Indoor Scenes with Articulated Objectscited by 7
  27. 2024Towards a copilot in BIM authoring tool using a large language model-based agent for intelligent human-machine interactioncited by 6
  28. 2025HIS-GPT: Towards 3D Human-In-Scene Multimodal Understandingcited by 6
  29. 2025From Programs to Poses: Factored Real-World Scene Generation via Learned Program Librariescited by 6
  30. 2026WorldClaw: Agentic 3D Open-World Generation at Scalecited by 6
  31. 2025ImmerseGen: Agent-Guided Immersive World Generation with Alpha-Textured Proxiescited by 5
  32. 2025LatticeWorld: A Multimodal Large Language Model-Empowered Framework for Interactive Complex World Generationcited by 5
  33. 2025MajutsuCity: Language-driven Aesthetic-adaptive City Generation with Controllable 3D Assets and Layoutscited by 5
  34. 2026SceneAssistant: A Visual Feedback Agent for Open-Vocabulary 3D Scene Generationcited by 5
  35. 2026BrickNet: Graph-Backed Generative Brick Assemblycited by 5
  36. 2026Imagine a City: CityGenAgent for Procedural 3D City Generationcited by 4
  37. 2026Mage: Multi-Axis Evaluation of LLM-Generated Executable Game Scenes Beyond Compile-Pass Ratecited by 4
  38. 2026SimWorlds: A Multi-Agent System for Dynamic 3D Scene Creationcited by 4

50 most cited of 91 All 91 (JSON)