mlsanityData to 14 Oct 2026

MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation

arXiv 2412.03558 ↗First posted 4 Dec 2024cs.CVcited by 117

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

This paper introduces MIDI, a novel paradigm for compositional 3D scene generation from a single image. Unlike existing methods that rely on reconstruction or retrieval techniques or recent approaches that employ multi-stage object-by-object generation, MIDI extends pre-trained image-to-3D object generation models to multi-instance diffusion models, enabling the simultaneous generation of multiple 3D instances with accurate spatial relationships and high generalizability. At its core, MIDI incorporates a novel multi-instance attention mechanism, that effectively captures inter-object interactions and spatial coherence directly within the generation process, without the need for complex multi-step processes. The method utilizes partial object images and global scene context as inputs, directly modeling object completion during 3D generation. During training, we effectively supervise the interactions between 3D instances using a limited amount of scene-level data, while incorporating single-object data for regularization, thereby maintaining the pre-trained generalization ability. MIDI demonstrates state-of-the-art performance in image-to-scene generation, validated through evaluations on synthetic data, real-world scene data, and stylized scene images generated by text-to-image diffusion models.

Key facts. Cited by 117 papers in the map (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 75 of its 84 references that are in the map. First posted on arXiv 4 Dec 2024.

Builds on

75 of 84 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024LT3SD: Latent Trees for 3D Scene Diffusion12 Sep
  2. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets30 May
  3. 2024Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single Image30 May
  4. 2024Part123: Part-aware 3D Reconstruction from a Single-view Image27 May
  5. 2024CraftsMan3D: High-fidelity Mesh Generation with 3D Native Generation and Interactive Geometry Refiner23 May
  6. 2024Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformer23 May
  7. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion18 Mar
  8. 2023EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusion11 Dec
  9. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion14 Nov
  10. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusion23 Oct
  11. 2023HiFi-123: Towards High-fidelity One Image to 3D Content Generation10 Oct
  12. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Image7 Sep
3 more
  1. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization29 Jun
  2. 2023DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis24 Mar
  3. 20233DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models26 Jan

Other references · 60

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  11. 2022Scalable Diffusion Models with Transformerscited by 6,054
  12. 2022Classifier-Free Diffusion Guidancecited by 5,847
38 more
  1. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  2. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  3. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  6. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  7. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  8. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,712
  9. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  10. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  11. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 929
  12. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 904
  13. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 849
  14. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 774
  15. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 739
  16. 2024InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Modelscited by 433
  17. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 385
  18. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 380
  19. 2021Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scanscited by 336
  20. 2021Neural RGB-D Surface Reconstructioncited by 313
  21. 20203D-FUTURE: 3D Furniture shape with TextUREcited by 311
  22. 2024GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generationcited by 301
  23. 2023Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformerscited by 258
  24. 2024TripoSR: Fast 3D Object Reconstruction from a Single Imagecited by 232
  25. 2023Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representationcited by 228
  26. 2024CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Modelcited by 215
  27. 2021ATISS: Autoregressive Transformers for Indoor Scene Synthesiscited by 200
  28. 2020Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Imagecited by 172
  29. 2021Holistic 3D Scene Understanding from a Single Image with Implicit Representationcited by 94
  30. 2024BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolationcited by 83
  31. 2016IM2CADcited by 78
  32. 2021Panoptic 3D Scene Reconstruction From a Single RGB Imagecited by 70
  33. 2021ROCA: Robust CAD Model Retrieval and Alignment from a Single Imagecited by 67
  34. 2020Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrievecited by 65
  35. 2024Gen3DSR: Generalizable 3D Scene Reconstruction via Divide and Conquer from a Single Viewcited by 61
  36. 2024ComboVerse: Compositional 3D Assets Creation Using Spatially-Aware Diffusion Guidancecited by 59
  37. 2023DiffCAD: Weakly-Supervised Probabilistic CAD Model Retrieval and Alignment from an RGB Imagecited by 48
  38. 2024REPARO: Compositional 3D Assets Generation with Differentiable 3D Layout Alignmentcited by 45

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 39; 2026: 78.

117 papers.

Sort:

By area:

  1. 2025SAM 3D: 3Dfy Anything in Imagescited by 247
  2. 2025CAST: Component-Aligned 3D Scene Reconstruction from an RGB Imagecited by 106
  3. 2025PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformerscited by 84
  4. 2025OmniPart: Part-Aware 3D Generation with Semantic Decoupling and Structural Cohesioncited by 61
  5. 2025SceneGen: Single-Image 3D Scene Generation in One Feedforward Passcited by 54
  6. 2025VoxHammer: Training-Free Precise and Coherent 3D Editing in Native 3D Spacecited by 45
  7. 2025AutoPartGen: Autogressive 3D Part Generation and Discoverycited by 33
  8. 2025SynCity: Training-Free Generation of 3D Worldscited by 31
  9. 20253D Scene Generation: A Surveycited by 24
  10. 2026ShapeR: Robust Conditional 3D Shape Generation from Casual Capturescited by 22
  11. 2025InternScenes: A Large-scale Simulatable Indoor Scene Dataset with Realistic Layoutscited by 20
  12. 2025WorldGen: From Text to Traversable and Interactive 3D Worldscited by 19
38 more
  1. 2025HiScene: Creating Hierarchical 3D Scenes with Isometric View Generationcited by 18
  2. 2025SceneMaker: Open-set 3D Scene Generation with Decoupled De-occlusion and Pose Estimation Modelcited by 17
  3. 2025MesaTask: Towards Task-Driven Tabletop Scene Generation via 3D Spatial Reasoningcited by 16
  4. 2025PAT3D: Physics-Augmented Text-to-3D Scene Generationcited by 16
  5. 2026SimRecon: SimReady Compositional Scene Reconstruction from Real Videoscited by 12
  6. 20263D-Fixer: Coarse-to-Fine In-place Completion for 3D Scenes from a Single Imagecited by 11
  7. 20253D-RE-GEN: 3D Reconstruction of Indoor Scenes with a Generative Frameworkcited by 10
  8. 2025PartSDF: Part-Based Implicit Neural Representation for Composite 3D Shape Parametrization and Optimizationcited by 9
  9. 2025CUPID: Generative 3D Reconstruction via Joint Object and Pose Modelingcited by 8
  10. 2025TRELLISWorld: Training-Free World Generation from Object Generatorscited by 8
  11. 2026ArtLLM: Generating Articulated Assets via 3D LLMcited by 8
  12. 2026WorldClaw: Agentic 3D Open-World Generation at Scalecited by 8
  13. 2025Digital Twin Generation from Visual Data: A Surveycited by 7
  14. 2025Video Perception Models for 3D Scene Synthesiscited by 7
  15. 2026Reconstruction by Generation: 3D Multi-Object Scene Reconstruction from Sparse Observationscited by 7
  16. 2025Towards Geometric and Textural Consistency 3D Scene Generation via Single Image-guided Model Generation and Layout Optimizationcited by 6
  17. 2025M3DLayout: A Multi-Source Dataset of 3D Indoor Layouts and Structured Descriptions for 3D Generationcited by 6
  18. 2025MajutsuCity: Language-driven Aesthetic-adaptive City Generation with Controllable 3D Assets and Layoutscited by 6
  19. 2025GENA3D: Generative Amodal 3D Modeling by Bridging 2D Priors and 3D Coherencecited by 6
  20. 2025I-Scene: 3D Instance Models are Implicit Generalizable Spatial Learnerscited by 6
  21. 2026SceneTransporter: Optimal Transport-Guided Compositional Latent Diffusion for Single-Image Structured 3D Scene Generationcited by 6
  22. 2026REST3D: Reconstructing Physically Stable 3D Scenes from a Single Imagecited by 6
  23. 2026One Video, One World: Turning Monocular Video into Physical 4D Scenescited by 6
  24. 2026Engine-Native Editable 3D World Reconstruction with Objects and Lightingcited by 6
  25. 2025ImmerseGen: Agent-Guided Immersive World Generation with Alpha-Textured Proxiescited by 5
  26. 2025ZeroScene: A Zero-Shot Framework for 3D Scene Generation from a Single Image and Controllable Texture Editingcited by 5
  27. 2025MoCA: Mixture-of-Components Attention for Scalable Compositional 3D Generationcited by 5
  28. 2026MonoArt: Progressive Structural Reasoning for Monocular Articulated 3D Reconstructioncited by 5
  29. 2026SimuScene: Simulation-Ready Compositional 3D Scene Reconstruction from a Single Imagecited by 5
  30. 2025HLG: Comprehensive 3D Room Construction via Hierarchical Layout Generationcited by 4
  31. 2026Simulation-Ready Cluttered Scene Estimation via Physics-aware Joint Shape and Pose Optimizationcited by 4
  32. 2026Points-to-3D: Structure-Aware 3D Generation with Point Cloud Priorscited by 4
  33. 2026SCMAPR: Self-Correcting Multi-Agent Prompt Refinement for Complex-Scenario Text-to-Video Generationcited by 4
  34. 2025IL3D: A Large-Scale Indoor Layout Dataset for LLM-Driven 3D Scene Generationcited by 3
  35. 2025GeoWorld: Providing Full-frame Geometry Features to Facilitate 3D Scene Generationcited by 3
  36. 2025TabletopGen: Tabletop Scene Generation and Interactive Simulation for Robotic Manipulationcited by 3
  37. 2026Research on World Models Is Not Merely Injecting World Knowledge into Specific Taskscited by 3
  38. 2026SceneLinker: Compositional 3D Scene Generation via Semantic Scene Graph from RGB Sequencescited by 3

50 most cited of 117 All 117 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Generating Compositional Scenes via Text-to-image RGBA Instance Generation
  2. 2025Recent Advances in 3D Object and Scene Generation: A Survey
  3. 2024Multi-view Image Prompted Multi-view Diffusion for Improved 3D Generation
  4. 2024PaintScene4D: Consistent 4D Scene Generation from Text Prompts
  5. 2024PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Models
  6. 2025Layout2Scene: 3D Semantic Layout Guided Scene Generation via Geometry and Appearance Diffusion Priors
  7. 2024Comp4D: LLM-Guided Compositional 4D Scene Generation
  8. 2024Draw Like an Artist: Complex Scene Generation with Diffusion Model via Composition, Painting, and Retouching