mlsanityData to 7 Oct 2026

SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code

arXiv 2403.01248 ↗First posted 2 Mar 2024cs.CV cs.AI cs.CLcited by 125

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

This paper introduces SceneCraft, a Large Language Model (LLM) Agent converting text descriptions into Blender-executable Python scripts which render complex scenes with up to a hundred 3D assets. This process requires complex spatial planning and arrangement. We tackle these challenges through a combination of advanced abstraction, strategic planning, and library learning. SceneCraft first models a scene graph as a blueprint, detailing the spatial relationships among assets in the scene. SceneCraft then writes Python scripts based on this graph, translating relationships into numerical constraints for asset layout. Next, SceneCraft leverages the perceptual strengths of vision-language foundation models like GPT-V to analyze rendered images and iteratively refine the scene. On top of this process, SceneCraft features a library learning mechanism that compiles common script functions into a reusable library, facilitating continuous self-improvement without expensive LLM parameter tuning. Our evaluation demonstrates that SceneCraft surpasses existing LLM-based agents in rendering complex scenes, as shown by its adherence to constraints and favorable human assessments. We also showcase the broader application potential of SceneCraft by reconstructing detailed 3D scenes from the Sintel movie and guiding a video generative model with generated scenes as intermediary control signal.

Key facts. Cited by 125 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 26 of its 38 references that are in the map. First posted on arXiv 2 Mar 2024.

Builds on

26 of 38 references.

References · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  5. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  6. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  7. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  8. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  9. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  10. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  11. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  12. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
14 more
  1. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  2. 2023Universal Self-Consistency for Large Language Model Generationcited by 198
  3. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 155
  4. 2023Infinite Photorealistic Worlds using Procedural Generationcited by 142
  5. 2020Unsupervised Question Decomposition for Question Answeringcited by 118
  6. 20233D-GPT: Procedural 3D Modeling with Large Language Modelscited by 91
  7. 2023LEGO-Net: Learning Regular Rearrangements of Objects in Roomscited by 75
  8. 2023Vision-Language Models as a Source of Rewardscited by 65
  9. 2023GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planningcited by 61
  10. 2023RoomDreamer: Text-Driven 3D Indoor Scene Synthesis with Coherent Geometry and Texturecited by 53
  11. 2015Text to 3D Scene Generation with Rich Lexical Groundingcited by 36
  12. 2017SceneSeer: 3D Scene Design with Natural Languagecited by 20
  13. 2023Advances in Data-Driven Analysis and Synthesis of 3D Indoor Scenescited by 19
  14. 2023Unsupervised Traffic Scene Generation with Synthetic 3D Scene Graphscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 46; 2026: 61.

125 papers.

Sort:

By area:

  1. 2025Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generationcited by 65
  2. 2025SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agentcited by 53
  3. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  4. 2024Chat2SVG: Vector Graphics Generation with Large Language Models and Image Diffusion Modelscited by 36
  5. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  6. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  7. 2024The Scene Language: Representing Scenes with Programs, Words, and Embeddingscited by 31
  8. 2025FilmAgent: A Multi-Agent Framework for End-to-End Film Automation in Virtual 3D Spacescited by 28
  9. 2024AutoVFX: Physically Realistic Video Editing from Natural Language Instructionscited by 27
  10. 2025BlenderGym: Benchmarking Foundational Model Systems for Graphics Editingcited by 27
  11. 2025SceneEval: Evaluating Semantic Coherence in Text-Conditioned 3D Indoor Scene Synthesiscited by 26
  12. 2025FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placementcited by 24
38 more
  1. 20253D Scene Generation: A Surveycited by 24
  2. 2024UrbanWorld: An Urban World Model for 3D City Generationcited by 23
  3. 2024ChatGarment: Garment Estimation, Generation and Editing via Large Language Modelscited by 22
  4. 2025HSM: Hierarchical Scene Motifs for Multi-Scale Indoor Scene Generationcited by 22
  5. 2024Chat2Layout: Interactive 3D Furniture Layout with a Multimodal LLMcited by 20
  6. 2025LL3M: Large Language 3D Modelerscited by 18
  7. 2025WorldGen: From Text to Traversable and Interactive 3D Worldscited by 18
  8. 2025IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Renderingcited by 17
  9. 2024Kubrick: Multimodal Agent Collaborations for Synthetic Video Generationcited by 16
  10. 2024EditRoom: LLM-parameterized Graph Diffusion for Composable 3D Room Layout Editingcited by 14
  11. 2025OptiScene: LLM-driven Indoor Scene Layout Generation via Scaled Human-aligned Data Synthesis and Multi-Stage Preference Optimizationcited by 14
  12. 2025HOLODECK 2.0: Vision-Language-Guided 3D World Generation with Editingcited by 14
  13. 2025SPATIALGEN: Layout-guided 3D Indoor Scene Generationcited by 14
  14. 2025Virtual Community: An Open World for Humans, Robots, and Societycited by 13
  15. 2026SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learningcited by 13
  16. 2026Code2Worlds: Empowering Coding LLMs for 4D World Generationcited by 12
  17. 2025BannerAgency: Advertising Banner Design with Multimodal LLM Agentscited by 11
  18. 2025MoVer: Motion Verification for Motion Graphics Animationscited by 10
  19. 2025BIMgent: Towards Autonomous Building Modeling via Computer-use Agentscited by 10
  20. 2025Follow-Your-Instruction: A Comprehensive MLLM Agent for World Data Synthesiscited by 10
  21. 2024Text2BIM: Generating Building Models Using a Large Language Model-based Multi-Agent Frameworkcited by 9
  22. 2025RoomCraft: Controllable and Complete 3D Indoor Scene Generationcited by 9
  23. 20263DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Codecited by 9
  24. 2024Reconstructing Animals and the Wildcited by 7
  25. 2024DI-PCG: Diffusion-based Efficient Inverse Procedural Content Generation for High-quality 3D Asset Creationcited by 7
  26. 2025BlenderFusion: 3D-Grounded Visual Editing and Generative Compositingcited by 7
  27. 2026VoxelCodeBench: Benchmarking 3D World Modeling Through Code Generationcited by 7
  28. 2026SceneCode: Executable World Programs for Editable Indoor Scenes with Articulated Objectscited by 7
  29. 2025Towards Geometric and Textural Consistency 3D Scene Generation via Single Image-guided Model Generation and Layout Optimizationcited by 6
  30. 2025SemLayoutDiff: Semantic Layout Generation with Diffusion Model for Indoor Scene Synthesiscited by 6
  31. 2026WorldClaw: Agentic 3D Open-World Generation at Scalecited by 6
  32. 2024ParSEL: Parameterized Shape Editing with Languagecited by 5
  33. 2025LatticeWorld: A Multimodal Large Language Model-Empowered Framework for Interactive Complex World Generationcited by 5
  34. 2025DisCo-Layout: Disentangling and Coordinating Semantic and Physical Refinement in a Multi-Agent Framework for 3D Indoor Layout Synthesiscited by 5
  35. 2025MajutsuCity: Language-driven Aesthetic-adaptive City Generation with Controllable 3D Assets and Layoutscited by 5
  36. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 5
  37. 2026SceneAssistant: A Visual Feedback Agent for Open-Vocabulary 3D Scene Generationcited by 5
  38. 2026BrickNet: Graph-Backed Generative Brick Assemblycited by 5

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026ViSculpt: Visual-Centric Agentic Geometry Editing
  2. 2026Code4Scene: Benchmarking Coding Agents for Constructing and Editing 3D Scenes
  3. 2025WorldCraft: Photo-Realistic 3D World Creation and Customization via LLM Agents
  4. 2024SceneGenAgent: Precise Industrial Scene Generation with Coding Agent
  5. 2026Articraft: An Agentic System for Scalable Articulated 3D Asset Generation
  6. 2024L3GO: Language Agents with Chain-of-3D-Thoughts for Generating Unconventional Objects
  7. 2026SceneMosaic: Efficient and Diverse Simulation-Ready Scene Generation via Hybrid Agentic Layout Evolution
  8. 2026DesignAgent3D: Interactive 3D Scene Editing via Designer-like Multimodal Reasoning