mlsanityData to 7 Oct 2026

MaGRITTe: Manipulative and Generative 3D Realization from Image, Topview and Text

arXiv 2404.00345 ↗First posted 30 Mar 2024cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

The generation of 3D scenes from user-specified conditions offers a promising avenue for alleviating the production burden in 3D applications. Previous studies required significant effort to realize the desired scene, owing to limited control conditions. We propose a method for controlling and generating 3D scenes under multimodal conditions using partial images, layout information represented in the top view, and text prompts. Combining these conditions to generate a 3D scene involves the following significant difficulties: (1) the creation of large datasets, (2) reflection on the interaction of multimodal conditions, and (3) domain dependence of the layout conditions. We decompose the process of 3D scene generation into 2D image generation from the given conditions and 3D scene generation from 2D images. 2D image generation is achieved by fine-tuning a pretrained text-to-image model with a small artificial dataset of partial images and layouts, and 3D scene generation is achieved by layout-conditioned depth estimation and neural radiance fields (NeRF), thereby avoiding the creation of large datasets. The use of a common representation of spatial information using 360-degree images allows for the consideration of multimodal condition interactions and reduces the domain dependence of the layout control. The experimental results qualitatively and quantitatively demonstrated that the proposed method can generate 3D scenes in diverse domains, from indoor to outdoor, according to multimodal conditions.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 58 of its 76 references that are in the map. First posted on arXiv 30 Mar 2024.

Builds on

58 of 76 references.

References · 58

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  11. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  12. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
38 more
  1. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  2. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  3. 2020AdaBins: Depth Estimation using Adaptive Binscited by 596
  4. 2016Deeper Depth Prediction with Fully Convolutional Residual Networkscited by 402
  5. 2019Structured3D: A Large Photo-realistic Dataset for Structured 3D Modelingcited by 285
  6. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  7. 2020Learning to Recover 3D Scene Shape from a Single Imagecited by 214
  8. 2023MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusioncited by 210
  9. 2017Learning to Predict Indoor Illumination from a Single Imagecited by 160
  10. 2023SceneScape: Text-Driven Consistent Scene Generationcited by 160
  11. 2022GAUDI: A Neural Architect for Immersive 3D Scene Generationcited by 136
  12. 2021Unconstrained Scene Generation with Locally Conditioned Radiance Fieldscited by 130
  13. 2021Transformer-Based Attention Networks for Continuous Pixel-Wise Predictioncited by 125
  14. 2017Semi-Supervised Deep Learning for Monocular Depth Map Predictioncited by 109
  15. 2020HoHoNet: 360 Indoor Holistic Understanding with Latent Horizontal Featurescited by 104
  16. 2023Learning to Render Novel Views from Wide-Baseline Stereo Pairscited by 103
  17. 2023BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layoutcited by 102
  18. 2018OmniDepth: Dense Depth Estimation for Indoors Spherical Panoramascited by 98
  19. 2018Structured Attention Guided Convolutional Neural Fields for Monocular Depth Estimationcited by 78
  20. 2023NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Modelscited by 76
  21. 2023SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collectionscited by 72
  22. 2017Multi-Scale Continuous CRFs as Sequential Deep Networks for Monocular Depth Estimationcited by 70
  23. 2023InfiniCity: Infinite-Scale City Synthesiscited by 68
  24. 2020Lighthouse: Predicting Lighting Volumes for Spatially-Coherent Illuminationcited by 62
  25. 2021360MonoDepth: High-Resolution 360deg Monocular Depth Estimationcited by 60
  26. 2023CC3D: Layout-Conditioned Generation of Compositional 3D Scenescited by 58
  27. 2019Neural Illumination: Lighting Prediction for Indoor Environmentscited by 57
  28. 2023ControlRoom3D: Room Generation using Semantic Proxy Roomscited by 54
  29. 2023LDM3D: Latent Diffusion Model for 3Dcited by 53
  30. 2019Spherical View Synthesis for Self-Supervised 360 Depth Estimationcited by 51
  31. 2022Diverse Plausible 360-Degree Image Outpainting for Efficient 3DCG Background Creationcited by 47
  32. 2022RGBD2: Generative Scene Synthesis via Incremental View Inpainting using RGBD Diffusion Modelscited by 45
  33. 2023PERF: Panoramic Neural Radiance Field from a Single Panoramacited by 45
  34. 2023Persistent Nature: A Generative Model of Unbounded 3D Worldscited by 44
  35. 2023360-Degree Panorama Generation from Few Unregistered NFoV Imagescited by 42
  36. 2023HRDFuse: Monocular 360degDepth Estimation by Collaboratively Learning Holistic-with-Regional Depth Distributionscited by 37
  37. 2018Eliminating the Blind Spot: Adapting 3D Object Detection and Monocular Depth Estimation to 360deg Panoramic Imagerycited by 34
  38. 2017Im2Pano3D: Extrapolating 360 Structure and Semantics Beyond the Field of Viewcited by 25

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025MMGDreamer: Mixed-Modality Graph for Geometry-Controllable 3D Indoor Scene Generationcited by 18
  2. 2025VideoPanda: Video Panoramic Diffusion with Multi-view Attentioncited by 13