mlsanityData to 7 Oct 2026

Chat2Layout: Interactive 3D Furniture Layout with a Multimodal LLM

arXiv 2407.21333 ↗First posted 31 Jul 2024cs.CVcited by 20

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Automatic furniture layout is long desired for convenient interior design. Leveraging the remarkable visual reasoning capabilities of multimodal large language models (MLLMs), recent methods address layout generation in a static manner, lacking the feedback-driven refinement essential for interactive user engagement. We introduce Chat2Layout, a novel interactive furniture layout generation system that extends the functionality of MLLMs into the realm of interactive layout design. To achieve this, we establish a unified vision-question paradigm for in-context learning, enabling seamless communication with MLLMs to steer their behavior without altering model weights. Within this framework, we present a novel training-free visual prompting mechanism. This involves a visual-text prompting technique that assist MLLMs in reasoning about plausible layout plans, followed by an Offline-to-Online search (O2O-Search) method, which automatically identifies the minimal set of informative references to provide exemplars for visual-text prompting. By employing an agent system with MLLMs as the core controller, we enable bidirectional interaction. The agent not only comprehends the 3D environment and user requirements through linguistic and visual perception but also plans tasks and reasons about actions to generate and arrange furniture within the virtual space. Furthermore, the agent iteratively updates based on visual feedback from execution results. Experimental results demonstrate that our approach facilitates language-interactive generation and arrangement for diverse and complex 3D furniture.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 58 of its 87 references that are in the map. First posted on arXiv 31 Jul 2024.

Builds on

58 of 87 references.

References · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  8. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  9. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  10. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  11. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  12. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
38 more
  1. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  2. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  4. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  5. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
  6. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  7. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 730
  8. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  9. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  10. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  11. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  12. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  13. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 482
  14. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 376
  15. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 375
  16. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 369
  17. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  18. 20203D-FUTURE: 3D Furniture shape with TextUREcited by 306
  19. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  20. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  21. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  22. 2023DreamBooth3D: Subject-Driven Text-to-3D Generationcited by 235
  23. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  24. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
  25. 2021ATISS: Autoregressive Transformers for Indoor Scene Synthesiscited by 195
  26. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  27. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  28. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  29. 2020SceneFormer: Indoor Scene Generation with Transformerscited by 154
  30. 2022NeRF-Art: Text-Driven Neural Radiance Fields Stylizationcited by 126
  31. 2024SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Codecited by 125
  32. 2024InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Priorcited by 110
  33. 2018Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Modelscited by 105
  34. 2021Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphscited by 83
  35. 2023Fine-Grained Visual Promptingcited by 81
  36. 2023AnyHome: Open-Vocabulary Generation of Structured and Textured 3D Homescited by 81
  37. 2023LEGO-Net: Learning Regular Rearrangements of Objects in Roomscited by 75
  38. 2018Human-centric Indoor Scene Synthesis Using Stochastic Grammarcited by 73

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 11; 2026: 8.

20 papers.

Sort:

By area:

  1. 2024Chat2SVG: Vector Graphics Generation with Large Language Models and Image Diffusion Modelscited by 36
  2. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 31
  3. 2025SceneEval: Evaluating Semantic Coherence in Text-Conditioned 3D Indoor Scene Synthesiscited by 26
  4. 2025HSM: Hierarchical Scene Motifs for Multi-Scale Indoor Scene Generationcited by 22
  5. 2025RoomCraft: Controllable and Complete 3D Indoor Scene Generationcited by 9
  6. 2025DisCo-Layout: Disentangling and Coordinating Semantic and Physical Refinement in a Multi-Agent Framework for 3D Indoor Layout Synthesiscited by 5
  7. 2025HLG: Comprehensive 3D Room Construction via Hierarchical Layout Generationcited by 4
  8. 2026MUSE: Agentic 3D Scene Authoring via Memory-Grounded Incremental Requirement Satisfactioncited by 3
  9. 20254DGS-Craft: Consistent and Interactive 4D Gaussian Splatting Editingcited by 2
  10. 2026CasLayout: Cascaded 3D Layout Diffusion for Indoor Scene Synthesis with Implicit Relation Modelingcited by 1
  11. 2026Agentic Designer: Progressive Multi-Agent Collaboration for Structure-Aware Interior Layout Generationcited by 1
  12. 2025HierRelTriple: Guiding Indoor Layout Generation with Hierarchical Relationship Triplet Lossescited by 0
8 more
  1. 2025Handle-based Mesh Deformation Guided By Vision Language Modelcited by 0
  2. 2025WordCraft: Interactive Artistic Typography with Attention Awareness and Noise Blendingcited by 0
  3. 2025Large Language Models and 3D Vision for Intelligent Robotic Perception and Autonomycited by 0
  4. 2026Design-MLLM: A Reinforcement Alignment Framework for Verifiable and Aesthetic Interior Designcited by 0
  5. 2026Architect-Ant: Editable Automatic Furnishing of Architectural Floor Planscited by 0
  6. 2026Text-Driven 3D Indoor Scene Synthesis in Non-Manhattan Environmentscited by 0
  7. 2026ThinkBLOX: 3D Indoor Scene Generation with Progressive Reasoningcited by 0
  8. 2026Learning an Interior Layout Policy in a Domain Specific Language Action Spacecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Intelligent Co-Design: An Interactive LLM Framework for Interior Spatial Design via Multi-Modal Agents
  2. 2024Layout Generation Agents with Large Language Models
  3. 2024LLplace: The 3D Indoor Scene Layout Generation and Editing via Large Language Model
  4. 2026Function2Scene: 3D Indoor Scene Layout from Functional Specifications
  5. 2025Direct Numerical Layout Generation for 3D Indoor Scene Synthesis via Spatial Reasoning
  6. 2025M3DLayout: A Multi-Source Dataset of 3D Indoor Layouts and Structured Descriptions for 3D Generation
  7. 2025CAL-RAG: Retrieval-Augmented Multi-Agent Generation for Content-Aware Layout Design
  8. 2024LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models