mlsanityData to 14 Oct 2026

ExCellGen: Fast, Controllable, Photorealistic 3D Scene Generation from a Single Real-World Exemplar

arXiv 2412.16253 ↗First posted 20 Dec 2024cs.CV cs.GRcited by 0

Not on the map. Why?

Abstract (arXiv)

Photorealistic 3D scene generation is challenging due to the scarcity of large-scale, high-quality real-world 3D datasets and complex workflows requiring specialized expertise for manual modeling. These constraints often result in slow iteration cycles, where each modification demands substantial effort, ultimately stifling creativity. We propose a fast, exemplar-driven framework for generating 3D scenes from a single casual input, such as handheld video or drone footage. Our method first leverages 3D Gaussian Splatting (3DGS) to robustly reconstruct input scenes with a high-quality 3D appearance model. We then train a per-scene Generative Cellular Automaton (GCA) to produce a sparse volume of featurized voxels, effectively amortizing scene generation while enabling controllability. A subsequent patch-based remapping step composites the complete scene from the exemplar's initial 3D Gaussian splats, successfully recovering the appearance statistics of the input scene. The entire pipeline can be trained in less than 10 minutes for each exemplar and generates scenes in 0.5-2 seconds. Our method enables interactive creation with full user control, and we showcase complex 3D generation results from real-world exemplars within a self-contained interactive GUI.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 59 of its 77 references that are in the map. First posted on arXiv 20 Dec 2024.

Builds on

59 of 77 references.

References · 59

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  4. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  5. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  6. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  7. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  8. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,041
  9. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,032
  10. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,613
  11. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  12. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
38 more
  1. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,290
  2. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  3. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 938
  4. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 770
  5. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 652
  6. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 612
  7. 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 575
  8. 2023LERF: Language Embedded Radiance Fieldscited by 543
  9. 2023Compact 3D Gaussian Representation for Radiance Fieldcited by 372
  10. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 371
  11. 2023Splatter Image: Ultra-Fast Single-View 3D Reconstructioncited by 364
  12. 2019SinGAN: Learning a Generative Model from a Single Natural Imagecited by 348
  13. 2023LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenescited by 231
  14. 2022NeRF-Editing: Geometry Editing of Neural Radiance Fieldscited by 217
  15. 2024WonderWorld: Interactive 3D Scene Generation from a Single Imagecited by 202
  16. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 196
  17. 2023XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchiescited by 189
  18. 2024Reducing the Memory Footprint of 3D Gaussian Splattingcited by 133
  19. 2023Compositional 3D Scene Generation using Locally Conditioned Diffusioncited by 101
  20. 2024BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolationcited by 83
  21. 2023CityDreamer: Compositional Generative Model of Unbounded 3D Citiescited by 76
  22. 2024SCube: Instant Large-Scale Scene Reconstruction using VoxSplatscited by 60
  23. 2022SinFusion: Training Diffusion Models on a Single Image or Videocited by 58
  24. 2023CC3D: Layout-Conditioned Generation of Compositional 3D Scenescited by 58
  25. 2023ControlRoom3D: Room Generation using Semantic Proxy Roomscited by 56
  26. 2023Control3D: Towards Controllable Text-to-3D Generationcited by 46
  27. 2022SinDiffusion: Learning a Diffusion Model from a Single Natural Imagecited by 42
  28. 2024Score Distillation via Reparametrized DDIMcited by 41
  29. 2024LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesiscited by 39
  30. 2024A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesiscited by 37
  31. 2024Disentangled 3D Scene Generation with Layout Learningcited by 33
  32. 2022Learning to Generate 3D Shapes from a Single Examplecited by 31
  33. 2023Sin3DM: Learning a Diffusion Model from a Single 3D Textured Shapecited by 29
  34. 2024Coin3D: Controllable and Interactive 3D Assets Generation with Proxy-Guided Conditioningcited by 26
  35. 2024SketchDream: Sketch-based Text-to-3D Generation and Editingcited by 25
  36. 2024SceneFactor: Factored Latent 3D Diffusion for Controllable 3D Scene Generationcited by 24
  37. 2021Learning to Generate 3D Shapes with Generative Cellular Automatacited by 22
  38. 2023CityGen: Infinite and Controllable City Layout Generationcited by 22

50 most cited of 59 All 59 (JSON)

Built on by

Not cited yet.