mlsanityData to 7 Oct 2026

ControlRoom3D: Room Generation using Semantic Proxy Rooms

arXiv 2312.05208 ↗First posted 8 Dec 2023cs.CVcited by 54

Not on the map. Why?

Abstract (arXiv)

Manually creating 3D environments for AR/VR applications is a complex process requiring expert knowledge in 3D modeling software. Pioneering works facilitate this process by generating room meshes conditioned on textual style descriptions. Yet, many of these automatically generated 3D meshes do not adhere to typical room layouts, compromising their plausibility, e.g., by placing several beds in one bedroom. To address these challenges, we present ControlRoom3D, a novel method to generate high-quality room meshes. Central to our approach is a user-defined 3D semantic proxy room that outlines a rough room layout based on semantic bounding boxes and a textual description of the overall room style. Our key insight is that when rendered to 2D, this 3D representation provides valuable geometric and semantic information to control powerful 2D models to generate 3D consistent textures and geometry that aligns well with the proxy room. Backed up by an extensive study including quantitative metrics and qualitative user evaluations, our method generates diverse and globally plausible 3D room meshes, thus empowering users to design 3D rooms effortlessly without specialized knowledge.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Builds on 51 of its 57 references that are in the map. First posted on arXiv 8 Dec 2023.

Builds on

51 of 57 references.

References · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  5. 2023Segment Anythingcited by 8,158
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
38 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  4. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  5. 2016Improved Techniques for Training GANscited by 2,938
  6. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  8. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  9. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  10. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  11. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  12. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
  13. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  14. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  15. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646
  16. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  17. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 593
  18. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  19. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
  20. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  21. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusioncited by 280
  22. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 278
  23. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  24. 2015YFCC100M: The New Data in Multimedia Researchcited by 234
  25. 2023MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusioncited by 210
  26. 2022DiffRF: Rendering-Guided 3D Radiance Field Diffusioncited by 204
  27. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  28. 2022ReCo: Region-Controlled Text-to-Image Generationcited by 177
  29. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
  30. 2023SceneScape: Text-Driven Consistent Scene Generationcited by 160
  31. 2022GAUDI: A Neural Architect for Immersive 3D Scene Generationcited by 136
  32. 2023Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fieldscited by 122
  33. 2023Compositional 3D Scene Generation using Locally Conditioned Diffusioncited by 101
  34. 2023Set-the-Scene: Global-Local Training for Generating Controllable NeRF Scenescited by 75
  35. 2023IT3D: Improved Text-to-3D Generation with Explicit View Synthesiscited by 63
  36. 2023Ctrl-Room: Controllable Text-to-3D Room Meshes Generation with Layout Constraintscited by 59
  37. 2023CC3D: Layout-Conditioned Generation of Compositional 3D Scenescited by 58
  38. 2023LDM3D: Latent Diffusion Model for 3Dcited by 53

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 25; 2026: 10.

54 papers.

Sort:

By area:

  1. 2024LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Modelscited by 94
  2. 2024BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolationcited by 81
  3. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 46
  4. 2024Open-Universe Indoor Scene Generation using LLM Program Synthesis and Uncurated Object Databasescited by 43
  5. 2024SceneCraft: Layout-Guided 3D Scene Generationcited by 40
  6. 2024Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusioncited by 39
  7. 2024Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpaintingcited by 37
  8. 20243D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucinationcited by 36
  9. 2024Mixed Diffusion for 3D Indoor Scene Synthesiscited by 30
  10. 2024A Comprehensive Survey on 3D Content Generationcited by 28
  11. 2025ArtiScene: Language-Driven Artistic 3D Scene Generation Through Image Intermediarycited by 27
  12. 2024Frankenstein: Generating Semantic-Compositional 3D Scenes in One Tri-Planecited by 26
38 more
  1. 2025WorldExplorer: Towards Generating Fully Navigable 3D Scenescited by 26
  2. 2024SceneFactor: Factored Latent 3D Diffusion for Controllable 3D Scene Generationcited by 24
  3. 2024Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from User's Casual Sketchescited by 21
  4. 2025SuperDec: 3D Scene Decomposition with Superquadric Primitivescited by 19
  5. 2024Pano2Room: Novel View Synthesis from a Single Indoor Panoramacited by 16
  6. 2025SPATIALGEN: Layout-guided 3D Indoor Scene Generationcited by 14
  7. 2025Virtual Community: An Open World for Humans, Robots, and Societycited by 13
  8. 2025Layout2Scene: 3D Semantic Layout Guided Scene Generation via Geometry and Appearance Diffusion Priorscited by 10
  9. 2024HouseCrafter: Lifting Floorplans to 3D Scenes with 2D Diffusion Modelcited by 9
  10. 2025VideoFrom3D: 3D Scene Video Generation via Complementary Image and Video Diffusion Modelscited by 8
  11. 2026World Reconstruction From Inconsistent Viewscited by 8
  12. 2025Video Perception Models for 3D Scene Synthesiscited by 7
  13. 2025LACONIC: A 3D Layout Adapter for Controllable Image Creationcited by 6
  14. 2025Towards Geometric and Textural Consistency 3D Scene Generation via Single Image-guided Model Generation and Layout Optimizationcited by 6
  15. 2025SemLayoutDiff: Semantic Layout Generation with Diffusion Model for Indoor Scene Synthesiscited by 6
  16. 2025M3DLayout: A Multi-Source Dataset of 3D Indoor Layouts and Structured Descriptions for 3D Generationcited by 5
  17. 2025MOSAIC: Generating Consistent, Privacy-Preserving Scenes from Multiple Depth Views in Multi-Room Environmentscited by 4
  18. 2025HLG: Comprehensive 3D Room Construction via Hierarchical Layout Generationcited by 4
  19. 2024SpaceBlender: Creating Context-Rich Collaborative Spaces Through Generative 3D Scene Blendingcited by 3
  20. 20253D-Generalist: Self-Improving Vision-Language-Action Models for Crafting 3D Worldscited by 3
  21. 2025Self-Evolving 3D Scene Generation from a Single Imagecited by 3
  22. 2026Edit-As-Act: Goal-Regressive Planning for Open-Vocabulary 3D Indoor Scene Editingcited by 3
  23. 2024MaGRITTe: Manipulative and Generative 3D Realization from Image, Topview and Textcited by 2
  24. 2024Prim2Room: Layout-Controllable Room Mesh Generation from Primitivescited by 2
  25. 2025Constructing a 3D Scene from a Single Imagecited by 2
  26. 2025DreamAnywhere: Object-Centric Panoramic 3D Scene Generationcited by 2
  27. 2025AutoScape: Geometry-Consistent Long-Horizon Scene Generationcited by 2
  28. 2026WorldMesh: Generating Navigable Multi-Room 3D Scenes via Mesh-Conditioned Image Diffusioncited by 2
  29. 2025Dream-to-Recon: Monocular 3D Reconstruction with Diffusion-Depth Distillation from Single Imagescited by 1
  30. 2025A Database-Driven Framework for 3D Level Generation with LLMscited by 1
  31. 2025MetaFind: Scene-Aware 3D Asset Retrieval for Coherent Metaverse Scene Generationcited by 1
  32. 2025MVRoom: Controllable 3D Indoor Scene Generation with Multi-View Diffusion Modelscited by 1
  33. 2026RoamScene3D: Immersive Text-to-3D Scene Generation via Adaptive Object-aware Roamingcited by 1
  34. 2026HOG-Layout: Hierarchical 3D Scene Generation, Optimization and Editing via Vision-Language Modelscited by 1
  35. 2026Scenix: Sparse-View 3D Scene Reconstruction via Executable Scene Programscited by 1
  36. 2024ExCellGen: Fast, Controllable, Photorealistic 3D Scene Generation from a Single Real-World Exemplarcited by 0
  37. 2025LPA3D: 3D Room-Level Scene Generation from In-the-Wild Imagescited by 0
  38. 2025Top2Pano: Learning to Generate Indoor Panoramas from Top-Down Viewcited by 0

50 most cited of 54 All 54 (JSON)