mlsanityData to 7 Oct 2026

The Scene Language: Representing Scenes with Programs, Words, and Embeddings

arXiv 2410.16770 ↗First posted 22 Oct 2024cs.CV cs.AIcited by 31

Not on the map. Why?

Abstract (arXiv)

We introduce the Scene Language, a visual scene representation that concisely and precisely describes the structure, semantics, and identity of visual scenes. It represents a scene with three key components: a program that specifies the hierarchical and relational structure of entities in the scene, words in natural language that summarize the semantic class of each entity, and embeddings that capture the visual identity of each entity. This representation can be inferred from pre-trained language models via a training-free inference technique, given text or image inputs. The resulting scene can be rendered into images using traditional, neural, or hybrid graphics renderers. Together, this forms a robust, automated system for high-quality 3D and 4D scene generation. Compared with existing representations like scene graphs, our proposed Scene Language generates complex scenes with higher fidelity, while explicitly modeling the scene structures to enable precise control and editing.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 34 of its 51 references that are in the map. First posted on arXiv 22 Oct 2024.

Builds on

34 of 51 references.

References · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2023Segment Anythingcited by 8,158
  9. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  11. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  12. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
22 more
  1. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  2. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  3. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  4. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  5. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  6. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  7. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  8. 2016Generative Visual Manipulation on the Natural Image Manifoldcited by 348
  9. 2018Image Generation from Scene Graphscited by 324
  10. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  11. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 184
  12. 2024SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Codecited by 125
  13. 2024MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesiscited by 123
  14. 2017CSGNet: Neural Shape Parser for Constructive Solid Geometrycited by 106
  15. 2024GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splattingcited by 104
  16. 2023GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphscited by 96
  17. 20233D-GPT: Procedural 3D Modeling with Large Language Modelscited by 91
  18. 2018Holistic 3D Scene Parsing and Reconstruction from a Single RGB Imagecited by 69
  19. 2019Learning to Infer and Execute 3D Shape Programscited by 68
  20. 2024L3GO: Language Agents with Chain-of-3D-Thoughts for Generating Unconventional Objectscited by 26
  21. 2023Creative Agents: Empowering Agents with Imagination for Creative Taskscited by 24
  22. 2024SceneMotifCoder: Example-driven Visual Program Learning for Generating 3D Object Arrangementscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 21.

31 papers.

Sort:

By area:

  1. 2024LLaMA-Mesh: Unifying 3D Mesh Generation with Language Modelscited by 87
  2. 2025ArtiScene: Language-Driven Artistic 3D Scene Generation Through Image Intermediarycited by 27
  3. 2025IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Renderingcited by 17
  4. 2025HOLODECK 2.0: Vision-Language-Guided 3D World Generation with Editingcited by 14
  5. 2025MoVer: Motion Verification for Motion Graphics Animationscited by 10
  6. 2025Part-X-MLLM: Part-aware 3D Multimodal Large Language Modelcited by 6
  7. 2026Thinking in Blender: Staged Executable Inverse Graphics with Vision-Language Modelscited by 6
  8. 2026Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoningcited by 6
  9. 2025ShapeLib: Designing a library of programmatic 3D shape abstractions with Large Language Modelscited by 5
  10. 2025ReSpace: Text-Driven Autoregressive 3D Indoor Scene Synthesis and Editingcited by 5
  11. 2026SceneAssistant: A Visual Feedback Agent for Open-Vocabulary 3D Scene Generationcited by 5
  12. 2025Human-in-the-Loop Local Corrections of 3D Scene Layouts via Infillingcited by 4
19 more
  1. 2026Imagine a City: CityGenAgent for Procedural 3D City Generationcited by 4
  2. 2025VULCAN: Tool-Augmented Multi Agents for Iterative 3D Object Arrangementcited by 3
  3. 20263D Primitives are a Spatial Language for VLMscited by 2
  4. 2026HDSL: A Hierarchical Domain-Specific Language for Structured 3D Indoor Scene Generation and Localized Editing with LLM Agentscited by 2
  5. 2026NERFIFY: A Multi-Agent Framework for Turning NeRF Papers into Codecited by 1
  6. 2026Self-Consistency for LLM-Based Motion Trajectory Generation and Verificationcited by 1
  7. 2026Deltaynamics: Language-Based Representation for Inferring Rigid-Body Dynamics From Videoscited by 1
  8. 2026Perceive-then-Plan: Layout-as-Policy for Monocular 3D Scene Layout Estimationcited by 1
  9. 2026Global-Local Monte Carlo Tree Search in Vision-Language Models for Text-to-3D Indoor Scene Generationcited by 1
  10. 2026ProcFunc: Function-Oriented Abstractions for Procedural 3D Generation in Pythoncited by 0
  11. 2026Function2Scene: 3D Indoor Scene Layout from Functional Specificationscited by 0
  12. 2026Hunyuan3D-Buffalo 1.0: A Unified Multimodal Model for Scalable 3D Generation, Understanding, and Editingcited by 0
  13. 2026aDSL: Agentic 3D Creation via Joint Agent-Program Designcited by 0
  14. 2026TailorCoPilot: Enabling Agentic Pattern Making with Version-Controlled State Trackingcited by 0
  15. 2026Visual General Intelligence: A White Papercited by 0
  16. 2026ESG: Generating Physically Consistent Dynamic 3D Scenes from Text Descriptionscited by 0
  17. 2026LEGO-Anything: Coding Agents for 3D Scene Reconstructioncited by 0
  18. 2026MatLoom: Layered Text-to-Material Generation in a Compact Program Spacecited by 0
  19. 2026LiteReality-Agent: An Agentic System for Interactable 3D Indoor Scene Reconstructioncited by 0