mlsanityData to 7 Oct 2026

LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models

arXiv 2412.02193 ↗First posted 3 Dec 2024cs.CV cs.AIcited by 94

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Spatial reasoning is a fundamental aspect of human cognition, enabling intuitive understanding and manipulation of objects in three-dimensional space. While foundation models demonstrate remarkable performance on some benchmarks, they still struggle with 3D reasoning tasks like arranging objects in space according to open-ended language instructions, particularly in dense and physically constrained environments. We introduce LayoutVLM, a framework and scene layout representation that exploits the semantic knowledge of Vision-Language Models (VLMs) and supports differentiable optimization to ensure physical plausibility. LayoutVLM employs VLMs to generate two mutually reinforcing representations from visually marked images, and a self-consistent decoding process to improve VLMs spatial planning. Our experiments show that LayoutVLM addresses the limitations of existing LLM and constraint-based approaches, producing physically plausible 3D layouts better aligned with the semantic intent of input language instructions. We also demonstrate that fine-tuning VLMs with the proposed scene layout representation extracted from existing scene datasets can improve their reasoning performance.

Key facts. Cited by 94 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 94 references that are in the map. First posted on arXiv 3 Dec 2024.

Builds on

33 of 94 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning18 Mar
  2. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  3. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov

Other references · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  5. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  6. 2019Distance-IoU Loss: Faster and Better Learning for Bounding Box Regressioncited by 454
  7. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  8. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  9. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  10. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 257
  11. 2021ATISS: Autoregressive Transformers for Indoor Scene Synthesiscited by 195
  12. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
18 more
  1. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 160
  2. 2024GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generationcited by 151
  3. 2019IoU Loss for 2D/3D Object Detectioncited by 128
  4. 2024InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Priorcited by 110
  5. 2024GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splattingcited by 104
  6. 2023Compositional 3D Scene Generation using Locally Conditioned Diffusioncited by 101
  7. 2024SceneScript: Reconstructing Scenes With An Autoregressive Structured Language Modelcited by 95
  8. 2024I-Design: Personalized LLM Interior Designercited by 87
  9. 2023AnyHome: Open-Vocabulary Generation of Structured and Textured 3D Homescited by 81
  10. 2023LEGO-Net: Learning Regular Rearrangements of Objects in Roomscited by 75
  11. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generationcited by 69
  12. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 56
  13. 2023ControlRoom3D: Room Generation using Semantic Proxy Roomscited by 54
  14. 2024Open-Universe Indoor Scene Generation using LLM Program Synthesis and Uncurated Object Databasescited by 43
  15. 20243D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucinationcited by 36
  16. 2024Disentangled 3D Scene Generation with Layout Learningcited by 33
  17. 2024Agent3D-Zero: An Agent for Zero-shot 3D Understandingcited by 29
  18. 2024Lay-A-Scene: Personalized 3D Object Arrangement Using Text-to-Image Priorscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 37; 2026: 57.

94 papers.

Sort:

By area:

  1. 2025Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generationcited by 65
  2. 2025SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agentcited by 53
  3. 2025MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaversecited by 50
  4. 2025SceneGen: Single-Image 3D Scene Generation in One Feedforward Passcited by 49
  5. 2026SAGE: Scalable Agentic 3D Scene Generation for Embodied AIcited by 40
  6. 2026SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenescited by 39
  7. 2025Direct Numerical Layout Generation for 3D Indoor Scene Synthesis via Spatial Reasoningcited by 27
  8. 2025SceneEval: Evaluating Semantic Coherence in Text-Conditioned 3D Indoor Scene Synthesiscited by 26
  9. 2025DecAlign: Hierarchical Cross-Modal Alignment for Decoupled Multimodal Representation Learningcited by 23
  10. 2025HSM: Hierarchical Scene Motifs for Multi-Scale Indoor Scene Generationcited by 22
  11. 2025IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Renderingcited by 17
  12. 2025OptiScene: LLM-driven Indoor Scene Layout Generation via Scaled Human-aligned Data Synthesis and Multi-Stage Preference Optimizationcited by 14
38 more
  1. 2025HOLODECK 2.0: Vision-Language-Guided 3D World Generation with Editingcited by 14
  2. 2025SPATIALGEN: Layout-guided 3D Indoor Scene Generationcited by 14
  3. 2025FloorplanQA: A Benchmark for Spatial Reasoning in LLMs using Structured Representationscited by 13
  4. 2025Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMscited by 10
  5. 2025PlaceIt3D: Language-Guided Object Placement in Real 3D Scenescited by 8
  6. 2025CoT-lized Diffusion: Let's Reinforce T2I Generation Step-by-stepcited by 8
  7. 2025Yo'City: Personalized and Boundless 3D Realistic City Scene Generation via Self-Critic Expansioncited by 8
  8. 2025Generative AI for Autonomous Driving: Frontiers and Opportunitiescited by 7
  9. 2025Video Perception Models for 3D Scene Synthesiscited by 7
  10. 2025MoMaGen: Generating Demonstrations under Soft and Hard Constraints for Multi-Step Bimanual Mobile Manipulationcited by 7
  11. 2026PhyScensis: Physics-Augmented LLM Agents for Complex Physical Scene Arrangementcited by 7
  12. 2026SceneCode: Executable World Programs for Editable Indoor Scenes with Articulated Objectscited by 7
  13. 2025Towards Geometric and Textural Consistency 3D Scene Generation via Single Image-guided Model Generation and Layout Optimizationcited by 6
  14. 2025SemLayoutDiff: Semantic Layout Generation with Diffusion Model for Indoor Scene Synthesiscited by 6
  15. 2026SceneReVis: A Self-Reflective Vision-Grounded Framework for 3D Indoor Scene Synthesis via Multi-turn RLcited by 6
  16. 2025ReSpace: Text-Driven Autoregressive 3D Indoor Scene Synthesis and Editingcited by 5
  17. 2025ImmerseGen: Agent-Guided Immersive World Generation with Alpha-Textured Proxiescited by 5
  18. 2025DisCo-Layout: Disentangling and Coordinating Semantic and Physical Refinement in a Multi-Agent Framework for 3D Indoor Layout Synthesiscited by 5
  19. 2026AnchoredDream: Zero-Shot 360deg Indoor Scene Generation from a Single View via Geometric Groundingcited by 5
  20. 2026SceneAssistant: A Visual Feedback Agent for Open-Vocabulary 3D Scene Generationcited by 5
  21. 2026Code-as-Room: Generating 3D Rooms from Top-Down View Images via Agentic Code Synthesiscited by 5
  22. 2025ScanEdit: Hierarchically-Guided Functional 3D Scan Editingcited by 4
  23. 2026Imagine a City: CityGenAgent for Procedural 3D City Generationcited by 4
  24. 20253D-Generalist: Self-Improving Vision-Language-Action Models for Crafting 3D Worldscited by 3
  25. 2025IL3D: A Large-Scale Indoor Layout Dataset for LLM-Driven 3D Scene Generationcited by 3
  26. 2025VULCAN: Tool-Augmented Multi Agents for Iterative 3D Object Arrangementcited by 3
  27. 2026Edit-As-Act: Goal-Regressive Planning for Open-Vocabulary 3D Indoor Scene Editingcited by 3
  28. 2026Repurposing 3D Generative Model for Autoregressive Layout Generationcited by 3
  29. 2026LychSim: A Controllable and Interactive Simulation Framework for Vision Researchcited by 3
  30. 2026MUSE: Agentic 3D Scene Authoring via Memory-Grounded Incremental Requirement Satisfactioncited by 3
  31. 2025Learning to Place Objects with Programs and Iterative Self Trainingcited by 2
  32. 2025TabletopGen: Tabletop Scene Generation and Interactive Simulation for Robotic Manipulationcited by 2
  33. 2025Reasoning Path and Latent State Analysis for Multi-view Visual Spatial Reasoning: A Cognitive Science Perspectivecited by 2
  34. 2026SceneNAT: Masked Generative Modeling for Language-Guided Indoor Scene Synthesiscited by 2
  35. 2026From Correspondence to Actions: Human-Like Multi-Image Spatial Reasoning in Multi-modal Large Language Modelscited by 2
  36. 2026Mind-of-Director: Multi-modal Agent-Driven Film Previsualization via Collaborative Decision-Makingcited by 2
  37. 2026WorldAgents: Can Foundation Image Models be Agents for 3D World Models?cited by 2
  38. 2026SceneCritic: A Symbolic Evaluator for 3D Indoor Scene Synthesiscited by 2

50 most cited of 94 All 94 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026R3L: Reasoning 3D Layouts from Relative Spatial Relations
  2. 20263D-Layout-R1: Structured Reasoning for Language-Instructed Spatial Editing
  3. 2026IDEAL-Bench: Indoor Dataset and Evaluation suite for Analyzing 3D Layout reasoning
  4. 2025S2-MLLM: Boosting Spatial Reasoning Capability of MLLMs for 3D Visual Grounding with Structural Guidance
  5. 2026Thinking in Structures: Evaluating Spatial Intelligence in Constraint-Governed Spaces
  6. 2026RoboLayout: Differentiable 3D Scene Generation for Embodied Agents
  7. 2025Abstract 3D Perception for Spatial Intelligence in Vision-Language Models
  8. 2026Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering