mlsanityData to 7 Oct 2026

WorldCraft: Photo-Realistic 3D World Creation and Customization via LLM Agents

arXiv 2502.15601 ↗First posted 21 Feb 2025cs.CV cs.AI cs.GRcited by 21

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Constructing photorealistic virtual worlds has applications across various fields, but it often requires the extensive labor of highly trained professionals to operate conventional 3D modeling software. To democratize this process, we introduce WorldCraft, a system where large language model (LLM) agents leverage procedural generation to create indoor and outdoor scenes populated with objects, allowing users to control individual object attributes and the scene layout using intuitive natural language commands. In our framework, a coordinator agent manages the overall process and works with two specialized LLM agents to complete the scene creation: ForgeIt, which integrates an ever-growing manual through auto-verification to enable precise customization of individual objects, and ArrangeIt, which formulates hierarchical optimization problems to achieve a layout that balances ergonomic and aesthetic considerations. Additionally, our pipeline incorporates a trajectory control agent, allowing users to animate the scene and operate the camera through natural language interactions. Our system is also compatible with off-the-shelf deep 3D generators to enrich scene assets. Through evaluations and comparisons with state-of-the-art methods, we demonstrate the versatility of WorldCraft, ranging from single-object customization to intricate, large-scale interior and exterior scene designs. This system empowers non-professionals to bring their creative visions to life.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 36 of its 66 references that are in the map. First posted on arXiv 21 Feb 2025.

Builds on

36 of 66 references.

References · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  6. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  7. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  8. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  9. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  10. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  11. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  12. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
24 more
  1. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 444
  2. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 376
  3. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  4. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Modelscited by 273
  5. 2021ATISS: Autoregressive Transformers for Indoor Scene Synthesiscited by 195
  6. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 161
  7. 2023DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesiscited by 159
  8. 2023WonderJourney: Going from Anywhere to Everywherecited by 157
  9. 2020SceneFormer: Indoor Scene Generation with Transformerscited by 154
  10. 2023Infinite Photorealistic Worlds using Procedural Generationcited by 142
  11. 2021Unconstrained Scene Generation with Locally Conditioned Radiance Fieldscited by 130
  12. 2023Self-correcting LLM-controlled Diffusion Modelscited by 112
  13. 2023Compositional 3D Scene Generation using Locally Conditioned Diffusioncited by 101
  14. 2021GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worldscited by 98
  15. 2023GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphscited by 96
  16. 2022InfiniteNature-Zero: Learning Perpetual View Generation of Natural Scenes from Single Imagescited by 82
  17. 2024DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splattingcited by 81
  18. 2023SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collectionscited by 72
  19. 2024DreamScene: 3D Gaussian-based Text-to-3D Scene Generation via Formation Pattern Samplingcited by 72
  20. 2023CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusioncited by 54
  21. 2024A Vision Check-up for Language Modelscited by 36
  22. 2024Disentangled 3D Scene Generation with Layout Learningcited by 33
  23. 2024ChatCam: Empowering Camera Control through Conversational AIcited by 15
  24. 2024Forest2Seq: Revitalizing Order Prior for Sequential Indoor Scene Synthesiscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 14.

21 papers.

Sort:

By area:

  1. 20253D Scene Generation: A Surveycited by 24
  2. 2025HSM: Hierarchical Scene Motifs for Multi-Scale Indoor Scene Generationcited by 22
  3. 2025LayerCraft: Enhancing Text-to-Image Generation with CoT Reasoning and Layered Object Integrationcited by 19
  4. 2025Sat2City: 3D City Generation from A Single Satellite Image with Cascaded Latent Diffusioncited by 15
  5. 2025ImmerseGen: Agent-Guided Immersive World Generation with Alpha-Textured Proxiescited by 5
  6. 2026SceneAssistant: A Visual Feedback Agent for Open-Vocabulary 3D Scene Generationcited by 5
  7. 2026Imagine a City: CityGenAgent for Procedural 3D City Generationcited by 4
  8. 2025EmboMatrix: A Scalable Training-Ground for Embodied Decision-Makingcited by 3
  9. 2025ShapeCraft: LLM Agents for Structured, Textured and Interactive 3D Modelingcited by 3
  10. 2026StoryBlender: Inter-Shot Consistent and Editable 3D Storyboard with Spatial-temporal Dynamicscited by 3
  11. 2026EZBlender: Efficient 3D Editing with Plan-and-ReAct Agentcited by 2
  12. 2026From Idea to Co-Creation: A Planner-Actor-Critic Framework for Agent Augmented 3D Modelingcited by 1
9 more
  1. 2026World Craft: Agentic Framework to Create Visualizable Worlds via Textcited by 1
  2. 2026Perceive-then-Plan: Layout-as-Policy for Monocular 3D Scene Layout Estimationcited by 1
  3. 2026CinemaTraj: Composing Atomic Camera Trajectories for 3D Scenes with LLM Agentscited by 1
  4. 2026GardenDesigner: Encoding Aesthetic Principles into Jiangnan Garden Construction via a Chain of Agentscited by 0
  5. 2026ProcFunc: Function-Oriented Abstractions for Procedural 3D Generation in Pythoncited by 0
  6. 2026HomeWorld: A Unified Floorplan-to-Furnished Framework for Generating Controllable, Densely Interactive Whole-Home Scenescited by 0
  7. 2026Look-Before-Move: Narrative-Grounded World Visual Attention in Dynamic 3D Story Worldscited by 0
  8. 2026IDEAL-Bench: Indoor Dataset and Evaluation suite for Analyzing 3D Layout reasoningcited by 0
  9. 2026To See a World in a Living Context: Unified Indoor-Outdoor Urban World Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LL3M: Large Language 3D Modelers
  2. 2024SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code
  3. 2025WorldGen: From Text to Traversable and Interactive 3D Worlds
  4. 2025LatticeWorld: A Multimodal Large Language Model-Empowered Framework for Interactive Complex World Generation
  5. 2026WorldAct: Activating Monolithic 3D Worlds into Interactive-Ready Object-Centric Scenes
  6. 2025CustomX: Unified Character, Action, and Scene Customization in Video World Models
  7. 2026Code2Games: Enabling Coding Agents for Gaming World Generation
  8. 2026GS-Agent: Creating 4D Physical Worlds With Generative Simulation