mlsanityData to 7 Oct 2026

SceneGenAgent: Precise Industrial Scene Generation with Coding Agent

arXiv 2410.21909 ↗First posted 29 Oct 2024cs.CL cs.LG cs.SEcited by 12

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

The modeling of industrial scenes is essential for simulations in industrial manufacturing. While large language models (LLMs) have shown significant progress in generating general 3D scenes from textual descriptions, generating industrial scenes with LLMs poses a unique challenge due to their demand for precise measurements and positioning, requiring complex planning over spatial arrangement. To address this challenge, we introduce SceneGenAgent, an LLM-based agent for generating industrial scenes through C# code. SceneGenAgent ensures precise layout planning through a structured and calculable format, layout verification, and iterative refinement to meet the quantitative requirements of industrial scenarios. Experiment results demonstrate that LLMs powered by SceneGenAgent exceed their original performance, reaching up to 81.0% success rate in real-world industrial scene generation tasks and effectively meeting most scene generation requirements. To further enhance accessibility, we construct SceneInstruct, a dataset designed for fine-tuning open-source LLMs to integrate into SceneGenAgent. Experiments show that fine-tuning open-source LLMs on SceneInstruct yields significant performance improvements, with Llama3.1-70B approaching the capabilities of GPT-4o. Our code and data are available at https://github.com/THUDM/SceneGenAgent .

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 28 of its 66 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

28 of 66 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent21 Sep

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2018Datasheets for Datasetscited by 1,371
  11. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  12. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
15 more
  1. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  2. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  3. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 375
  4. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  5. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  6. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 257
  7. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  8. 2019Meta-Sim: Learning to Generate Synthetic Datasetscited by 128
  9. 2024GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splattingcited by 104
  10. 2023Compositional 3D Scene Generation using Locally Conditioned Diffusioncited by 101
  11. 2021SceneGen: Learning to Generate Realistic Traffic Scenescited by 83
  12. 2023Set-the-Scene: Global-Local Training for Generating Controllable NeRF Scenescited by 75
  13. 2023CompoNeRF: Text-guided Multi-object Compositional NeRF with Editable 3D Scene Layoutcited by 57
  14. 2023RoomDreamer: Text-Driven 3D Indoor Scene Synthesis with Coherent Geometry and Texturecited by 53
  15. 2017SceneSeer: 3D Scene Design with Natural Languagecited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 7.

12 papers.

  1. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  2. 2025HSM: Hierarchical Scene Motifs for Multi-Scale Indoor Scene Generationcited by 22
  3. 2025TDRM: Smooth Reward Models with Temporal Difference for LLM RL and Inferencecited by 8
  4. 2026MiniAppBench: Evaluating the Shift from Text to Interactive HTML Responses in LLM-Powered Assistantscited by 7
  5. 2025Can Large Language Models Master Complex Card Games?cited by 4
  6. 2026FlowScene: Style-Consistent Indoor Scene Generation with Multimodal Graph Rectified Flowcited by 4
  7. 2026DRAFT: Task Decoupled Latent Reasoning for Agent Safetycited by 1
  8. 2025Phythesis: Physics-Guided Evolutionary Scene Synthesis for Energy-Efficient Data Center Design via LLMscited by 0
  9. 2026Re2MoGen: Open-Vocabulary Motion Generation via LLM Reasoning and Physics-Aware Refinementcited by 0
  10. 2026Text-Driven 3D Indoor Scene Synthesis in Non-Manhattan Environmentscited by 0
  11. 2026Continual Learning in Transitioncited by 0
  12. 2026LEGO-Anything: Coding Agents for 3D Scene Reconstructioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code
  2. 2024Open-Universe Indoor Scene Generation using LLM Program Synthesis and Uncurated Object Databases
  3. 2026SceneOrchestra: Efficient Agentic 3D Scene Synthesis via Full Tool-Call Trajectory Generation
  4. 2026NaLA: A 3D Native LLM Layout Agent for High-quality 3D Scene Generation
  5. 2023Towards Language-guided Interactive 3D Generation: LLMs as Layout Interpreter with Generative Feedback
  6. 2025SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent
  7. 2026HDSL: A Hierarchical Domain-Specific Language for Structured 3D Indoor Scene Generation and Localized Editing with LLM Agents
  8. 2025Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generation