# SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code

arXiv: https://arxiv.org/abs/2403.01248 · HTML: https://mlsanity.com/paper/2403.01248/

## Key facts

- Cited by 134 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 26 of its 38 references that are in the map.
- First posted on arXiv 2 Mar 2024.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [VideoPoet: A Large Language Model for Zero-Shot Video Generation](https://mlsanity.com/paper/2312.14125/) (2023)
- [GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions](https://mlsanity.com/paper/2104.14806/) (2021)
- [Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs](https://mlsanity.com/paper/2401.11708/) (2024)
- [Universal Self-Consistency for Large Language Model Generation](https://mlsanity.com/paper/2311.17311/) (2023)
- [Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning](https://mlsanity.com/paper/2310.12921/) (2023)
- [Infinite Photorealistic Worlds using Procedural Generation](https://mlsanity.com/paper/2306.09310/) (2023)
- [Unsupervised Question Decomposition for Question Answering](https://mlsanity.com/paper/2002.09758/) (2020)
- [3D-GPT: Procedural 3D Modeling with Large Language Models](https://mlsanity.com/paper/2310.12945/) (2023)
- [LEGO-Net: Learning Regular Rearrangements of Objects in Rooms](https://mlsanity.com/paper/2301.09629/) (2023)
- [Vision-Language Models as a Source of Rewards](https://mlsanity.com/paper/2312.09187/) (2023)
- [GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planning](https://mlsanity.com/paper/2311.12631/) (2023)
- [RoomDreamer: Text-Driven 3D Indoor Scene Synthesis with Coherent Geometry and Texture](https://mlsanity.com/paper/2305.11337/) (2023)
- [Text to 3D Scene Generation with Rich Lexical Grounding](https://mlsanity.com/paper/1505.06289/) (2015)
- [SceneSeer: 3D Scene Design with Natural Language](https://mlsanity.com/paper/1703.00050/) (2017)
- [Advances in Data-Driven Analysis and Synthesis of 3D Indoor Scenes](https://mlsanity.com/paper/2304.03188/) (2023)
- [Unsupervised Traffic Scene Generation with Synthetic 3D Scene Graphs](https://mlsanity.com/paper/2303.08473/) (2023)

## Built on by (most cited first, 30 of 134)

- [Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generation](https://mlsanity.com/paper/2505.02836/) (2025)
- [SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent](https://mlsanity.com/paper/2509.20414/) (2025)
- [When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models](https://mlsanity.com/paper/2405.10255/) (2024)
- [Chat2SVG: Vector Graphics Generation with Large Language Models and Image Diffusion Models](https://mlsanity.com/paper/2411.16602/) (2024)
- [The Scene Language: Representing Scenes with Programs, Words, and Embeddings](https://mlsanity.com/paper/2410.16770/) (2024)
- [LLMs Meet Multimodal Generation and Editing: A Survey](https://mlsanity.com/paper/2405.19334/) (2024)
- [How Far Are We From AGI: Are LLMs All We Need?](https://mlsanity.com/paper/2405.10313/) (2024)
- [BlenderGym: Benchmarking Foundational Model Systems for Graphics Editing](https://mlsanity.com/paper/2504.01786/) (2025)
- [FilmAgent: A Multi-Agent Framework for End-to-End Film Automation in Virtual 3D Spaces](https://mlsanity.com/paper/2501.12909/) (2025)
- [AutoVFX: Physically Realistic Video Editing from Natural Language Instructions](https://mlsanity.com/paper/2411.02394/) (2024)
- [SceneEval: Evaluating Semantic Coherence in Text-Conditioned 3D Indoor Scene Synthesis](https://mlsanity.com/paper/2503.14756/) (2025)
- [FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placement](https://mlsanity.com/paper/2503.04919/) (2025)
- [UrbanWorld: An Urban World Model for 3D City Generation](https://mlsanity.com/paper/2407.11965/) (2024)
- [HSM: Hierarchical Scene Motifs for Multi-Scale Indoor Scene Generation](https://mlsanity.com/paper/2503.16848/) (2025)
- [3D Scene Generation: A Survey](https://mlsanity.com/paper/2505.05474/) (2025)
- [ChatGarment: Garment Estimation, Generation and Editing via Large Language Models](https://mlsanity.com/paper/2412.17811/) (2024)
- [Chat2Layout: Interactive 3D Furniture Layout with a Multimodal LLM](https://mlsanity.com/paper/2407.21333/) (2024)
- [LL3M: Large Language 3D Modelers](https://mlsanity.com/paper/2508.08228/) (2025)
- [IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering](https://mlsanity.com/paper/2506.23329/) (2025)
- [WorldGen: From Text to Traversable and Interactive 3D Worlds](https://mlsanity.com/paper/2511.16825/) (2025)
- [Code2Worlds: Empowering Coding LLMs for 4D World Generation](https://mlsanity.com/paper/2602.11757/) (2026)
- [SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning](https://mlsanity.com/paper/2605.09423/) (2026)
- [Kubrick: Multimodal Agent Collaborations for Synthetic Video Generation](https://mlsanity.com/paper/2408.10453/) (2024)
- [EditRoom: LLM-parameterized Graph Diffusion for Composable 3D Room Layout Editing](https://mlsanity.com/paper/2410.12836/) (2024)
- [HOLODECK 2.0: Vision-Language-Guided 3D World Generation with Editing](https://mlsanity.com/paper/2508.05899/) (2025)
- [3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code](https://mlsanity.com/paper/2606.01057/) (2026)
- [OptiScene: LLM-driven Indoor Scene Layout Generation via Scaled Human-aligned Data Synthesis and Multi-Stage Preference Optimization](https://mlsanity.com/paper/2506.07570/) (2025)
- [SPATIALGEN: Layout-guided 3D Indoor Scene Generation](https://mlsanity.com/paper/2509.14981/) (2025)
- [Virtual Community: An Open World for Humans, Robots, and Society](https://mlsanity.com/paper/2508.14893/) (2025)
- [BannerAgency: Advertising Banner Design with Multimodal LLM Agents](https://mlsanity.com/paper/2503.11060/) (2025)

## Abstract (arXiv)

This paper introduces SceneCraft, a Large Language Model (LLM) Agent converting text descriptions into Blender-executable Python scripts which render complex scenes with up to a hundred 3D assets. This process requires complex spatial planning and arrangement. We tackle these challenges through a combination of advanced abstraction, strategic planning, and library learning. SceneCraft first models a scene graph as a blueprint, detailing the spatial relationships among assets in the scene. SceneCraft then writes Python scripts based on this graph, translating relationships into numerical constraints for asset layout. Next, SceneCraft leverages the perceptual strengths of vision-language foundation models like GPT-V to analyze rendered images and iteratively refine the scene. On top of this process, SceneCraft features a library learning mechanism that compiles common script functions into a reusable library, facilitating continuous self-improvement without expensive LLM parameter tuning. Our evaluation demonstrates that SceneCraft surpasses existing LLM-based agents in rendering complex scenes, as shown by its adherence to constraints and favorable human assessments. We also showcase the broader application potential of SceneCraft by reconstructing detailed 3D scenes from the Sintel movie and guiding a video generative model with generated scenes as intermediary control signal.