mlsanityData to 14 Oct 2026

Lay-A-Scene: Personalized 3D Object Arrangement Using Text-to-Image Priors

arXiv 2406.00687 ↗First posted 2 Jun 2024cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

Generating 3D visual scenes is at the forefront of visual generative AI, but current 3D generation techniques struggle with generating scenes with multiple high-resolution objects. Here we introduce Lay-A-Scene, which solves the task of Open-set 3D Object Arrangement, effectively arranging unseen objects. Given a set of 3D objects, the task is to find a plausible arrangement of these objects in a scene. We address this task by leveraging pre-trained text-to-image models. We personalize the model and explain how to generate images of a scene that contains multiple predefined objects without neglecting any of them. Then, we describe how to infer the 3D poses and arrangement of objects from a 2D generated image by finding a consistent projection of objects onto the 2D scene. We evaluate the quality of Lay-A-Scene using 3D objects from Objaverse and human raters and find that it often generates coherent and feasible 3D object arrangements.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 39 of its 63 references that are in the map. First posted on arXiv 2 Jun 2024.

Builds on

39 of 63 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  5. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  6. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  7. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  8. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  9. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
  10. 2018Demystifying MMD GANscited by 1,293
  11. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,290
  12. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 702
27 more
  1. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  2. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  3. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  4. 2023Emergent Correspondence from Image Diffusioncited by 446
  5. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 334
  6. 2023RealFusion: 360deg Reconstruction of Any Object from a Single Imagecited by 306
  7. 2023TEXTure: Text-Guided Texturing of 3D Shapescited by 304
  8. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  9. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  10. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 231
  11. 2021ATISS: Autoregressive Transformers for Indoor Scene Synthesiscited by 200
  12. 2023Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignmentcited by 171
  13. 2022NeRF-Art: Text-Driven Neural Radiance Fields Stylizationcited by 126
  14. 2019LayoutVAE: Stochastic Scene Layout Generation From a Label Setcited by 119
  15. 2024InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Priorcited by 117
  16. 2021Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphscited by 84
  17. 2022TANGO: Text-driven Photorealistic and Robust 3D Stylization via Lighting Decompositioncited by 83
  18. 2023LEGO-Net: Learning Regular Rearrangements of Objects in Roomscited by 76
  19. 2020Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generationcited by 58
  20. 2023CC3D: Layout-Conditioned Generation of Compositional 3D Scenescited by 58
  21. 2022Understanding Pure CLIP Guidance for Voxel Grid NeRF Modelscited by 56
  22. 2023RoomDreamer: Text-Driven 3D Indoor Scene Synthesis with Coherent Geometry and Texturecited by 53
  23. 2017Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scenecited by 50
  24. 2021Indoor Scene Generation from a Collection of Semantic-Segmented Depth Imagescited by 27
  25. 2021LUMINOUS: Indoor Scene Generation for Embodied AI Challengescited by 12
  26. 20223D-TOGO: Towards Text-Guided Cross-Category 3D Object Generationcited by 8
  27. 2023iDesigner: A High-Resolution and Complex-Prompt Following Text-to-Image Diffusion Model for Interior Designcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 2.

7 papers.

  1. 2024LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Modelscited by 96
  2. 2025MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaversecited by 51
  3. 2025Towards Geometric and Textural Consistency 3D Scene Generation via Single Image-guided Model Generation and Layout Optimizationcited by 6
  4. 2026AnchoredDream: Zero-Shot 360deg Indoor Scene Generation from a Single View via Geometric Groundingcited by 5
  5. 2025ScanEdit: Hierarchically-Guided Functional 3D Scan Editingcited by 4
  6. 20253D-Generalist: Self-Improving Vision-Language-Action Models for Crafting 3D Worldscited by 3
  7. 2026TIMI: Training-Free Image-to-3D Multi-Instance Generation with Spatial Fidelitycited by 0