# SPSG: Self-Supervised Photometric Scene Generation from RGB-D Scans

arXiv: https://arxiv.org/abs/2006.14660 · HTML: https://mlsanity.com/paper/2006.14660/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jun 2020.

## Built on by (most cited first, 30 of 34)

- [Neural RGB-D Surface Reconstruction](https://mlsanity.com/paper/2104.04532/) (2021)
- [Text2Tex: Text-driven Texture Synthesis via Diffusion Models](https://mlsanity.com/paper/2303.11396/) (2023)
- [4DComplete: Non-Rigid Motion Estimation Beyond the Observable Surface](https://mlsanity.com/paper/2105.01905/) (2021)
- [Texturify: Generating Textures on 3D Shape Surfaces](https://mlsanity.com/paper/2204.02411/) (2022)
- [StyleMesh: Style Transfer for Indoor 3D Scene Reconstructions](https://mlsanity.com/paper/2112.01530/) (2021)
- [LT3SD: Latent Trees for 3D Scene Diffusion](https://mlsanity.com/paper/2409.08215/) (2024)
- [Watch Your Steps: Local Image and Scene Editing by Text Instructions](https://mlsanity.com/paper/2308.08947/) (2023)
- [SceneTex: High-Quality Texture Synthesis for Indoor Scenes via Diffusion Priors](https://mlsanity.com/paper/2311.17261/) (2023)
- [Reference-guided Controllable Inpainting of Neural Radiance Fields](https://mlsanity.com/paper/2304.09677/) (2023)
- [DiffComplete: Diffusion-based Generative 3D Shape Completion](https://mlsanity.com/paper/2306.16329/) (2023)
- [Mesh2Tex: Generating Mesh Textures from Image Queries](https://mlsanity.com/paper/2304.05868/) (2023)
- [Paris-CARLA-3D: A Real and Synthetic Outdoor Point Cloud Dataset for Challenging Tasks in 3D Mapping](https://mlsanity.com/paper/2111.11348/) (2021)
- [ShapeR: Robust Conditional 3D Shape Generation from Casual Captures](https://mlsanity.com/paper/2601.11514/) (2026)
- [Point Scene Understanding via Disentangled Instance Mesh Reconstruction](https://mlsanity.com/paper/2203.16832/) (2022)
- [Multi-sensor large-scale dataset for multi-view 3D reconstruction](https://mlsanity.com/paper/2203.06111/) (2022)
- [SC-Diff: 3D Shape Completion with Latent Diffusion Models](https://mlsanity.com/paper/2403.12470/) (2024)
- [Enhancement of Novel View Synthesis Using Omnidirectional Image Completion](https://mlsanity.com/paper/2203.09957/) (2022)
- [CompNVS: Novel View Synthesis with Scene Completion](https://mlsanity.com/paper/2207.11467/) (2022)
- [Clutter Detection and Removal in 3D Scenes with View-Consistent Inpainting](https://mlsanity.com/paper/2304.03763/) (2023)
- [TUVF: Learning Generalizable Texture UV Radiance Fields](https://mlsanity.com/paper/2305.03040/) (2023)
- [SSR-2D: Semantic 3D Scene Reconstruction from 2D Images](https://mlsanity.com/paper/2302.03640/) (2023)
- [Neural Part Priors: Learning to Optimize Part-Based Object Completion in RGB-D Scans](https://mlsanity.com/paper/2203.09375/) (2022)
- [Free-form 3D Scene Inpainting with Dual-stream GAN](https://mlsanity.com/paper/2212.08464/) (2022)
- [NeuralRoom: Geometry-Constrained Neural Implicit Surfaces for Indoor Scene Reconstruction](https://mlsanity.com/paper/2210.06853/) (2022)
- [CIRCLE: Convolutional Implicit Reconstruction and Completion for Large-scale Indoor Scene](https://mlsanity.com/paper/2111.12905/) (2021)
- [Efficient and Effective Model Extraction](https://mlsanity.com/paper/2409.14122/) (2024)
- [UV-free Texture Generation with Denoising and Geodesic Heat Diffusions](https://mlsanity.com/paper/2408.16762/) (2024)
- [SparseGNV: Generating Novel Views of Indoor Scenes with Sparse Input Views](https://mlsanity.com/paper/2305.07024/) (2023)
- [Automatic Scene Generation: State-of-the-Art Techniques, Models, Datasets, Challenges, and Future Prospects](https://mlsanity.com/paper/2410.01816/) (2024)
- [Unsigned Distance Field as an Accurate 3D Scene Representation for Neural Scene Completion](https://mlsanity.com/paper/2203.09167/) (2022)

## Abstract (arXiv)

We present SPSG, a novel approach to generate high-quality, colored 3D models of scenes from RGB-D scan observations by learning to infer unobserved scene geometry and color in a self-supervised fashion. Our self-supervised approach learns to jointly inpaint geometry and color by correlating an incomplete RGB-D scan with a more complete version of that scan. Notably, rather than relying on 3D reconstruction losses to inform our 3D geometry and color reconstruction, we propose adversarial and perceptual losses operating on 2D renderings in order to achieve high-resolution, high-quality colored reconstructions of scenes. This exploits the high-resolution, self-consistent signal from individual raw RGB-D frames, in contrast to fused 3D reconstructions of the frames which exhibit inconsistencies from view-dependent effects, such as color balancing or pose inconsistencies. Thus, by informing our 3D scene generation directly through 2D signal, we produce high-quality colored reconstructions of 3D scenes, outperforming state of the art on both synthetic and real data.