mlsanityData to 14 Oct 2026

RetrievalFuse: Neural 3D Scene Reconstruction with a Database

arXiv 2104.00024 ↗First posted 31 Mar 2021cs.CVcited by 28

Not on the map. Why?

Abstract (arXiv)

3D reconstruction of large scenes is a challenging problem due to the high-complexity nature of the solution space, in particular for generative neural networks. In contrast to traditional generative learned models which encode the full generative process into a neural network and can struggle with maintaining local details at the scene level, we introduce a new method that directly leverages scene geometry from the training database. First, we learn to synthesize an initial estimate for a 3D scene, constructed by retrieving a top-k set of volumetric chunks from the scene database. These candidates are then refined to a final scene generation with an attention-based refinement that can effectively select the most consistent set of geometry from the candidates and combine them together to create an output scene, facilitating transfer of coherent structures and local detail from train scene geometry. We demonstrate our neural scene reconstruction with a database for the tasks of 3D super resolution and surface reconstruction from sparse point clouds, showing that our approach enables generation of more coherent, accurate 3D scenes, improving on average by over 8% in IoU over state-of-the-art scene reconstruction.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 32 of its 43 references that are in the map. First posted on arXiv 31 Mar 2021.

Builds on

32 of 43 references.

References · 32

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  6. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  7. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,041
  8. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  9. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 979
  10. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  11. 2020Convolutional Occupancy Networkscited by 761
  12. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 651
20 more
  1. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 588
  2. 2017Fine-tuning CNN Image Retrieval with No Human Annotationcited by 499
  3. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 488
  4. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 380
  5. 2020Local Implicit Grid Representations for 3D Scenescited by 373
  6. 2020Implicit Functions in Feature Space for 3D Shape Reconstruction and Completioncited by 343
  7. 2020Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstructioncited by 321
  8. 2019What Do Single-view 3D Reconstruction Networks Learn?cited by 260
  9. 2016Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesiscited by 219
  10. 2017Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputscited by 189
  11. 2018Scan2CAD: Learning CAD Model Alignment in RGB-D Scanscited by 142
  12. 2018Neural Nearest Neighbors Networkscited by 98
  13. 2017Hierarchical Surface Prediction for 3D Object Reconstructioncited by 89
  14. 2016IM2CADcited by 78
  15. 2019SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scanscited by 77
  16. 2020Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrievecited by 65
  17. 2018Scan2Mesh: From Unstructured Range Scans to 3D Meshescited by 51
  18. 2020RetrieveGAN: Image Synthesis via Differentiable Patch Retrievalcited by 35
  19. 2020Texture Memory-Augmented Deep Patch-Based Image Inpaintingcited by 22
  20. 2018Scene Recomposition by Learning-based ICPcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 14; 2023: 4; 2024: 5; 2025: 2; 2026: 3.

28 papers.

Sort:

By area:

  1. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generatorcited by 210
  2. 2022KNN-Diffusion: Image Generation via Large-Scale Retrievalcited by 137
  3. 2022POCO: Point Convolution for Surface Reconstructioncited by 92
  4. 2022Texturify: Generating Textures on 3D Shape Surfacescited by 65
  5. 2022Text-Guided Synthesis of Artistic Images with Retrieval-Augmented Diffusion Modelscited by 64
  6. 2022Reconstructing Surfaces for Sparse Point Clouds with On-Surface Priorscited by 60
  7. 2024LT3SD: Latent Trees for 3D Scene Diffusioncited by 55
  8. 2022Surface Reconstruction from Point Clouds by Learning Predictive Context Priorscited by 54
  9. 2023RoomDreamer: Text-Driven 3D Indoor Scene Synthesis with Coherent Geometry and Texturecited by 53
  10. 2022Retrieval-Enhanced Machine Learningcited by 47
  11. 2022Surface Reconstruction from Point Clouds: A Survey and a Benchmarkcited by 41
  12. 2022Semi-Parametric Neural Image Synthesiscited by 37
16 more
  1. 2022Memories are One-to-Many Mapping Alleviators in Talking Face Generationcited by 22
  2. 2026ShapeR: Robust Conditional 3D Shape Generation from Casual Capturescited by 22
  3. 2023VDN-NeRF: Resolving Shape-Radiance Ambiguity via View-Dependence Normalizationcited by 15
  4. 2024Nearest Neighbour Score Estimators for Diffusion Generative Modelscited by 10
  5. 2022What's the Situation with Intelligent Mesh Generation: A Survey and Perspectivescited by 9
  6. 2022Probabilistic Implicit Scene Completioncited by 8
  7. 2022PatchRD: Detail-Preserving Shape Completion by Learning Patch Retrieval and Deformationcited by 7
  8. 2024DECOLLAGE: 3D Detailization by Controllable, Localized, and Learned Geometry Enhancementcited by 7
  9. 2024SS3DM: Benchmarking Street-View Surface Reconstruction with a Synthetic 3D Mesh Datasetcited by 4
  10. 2023A Review of Deep Learning-Powered Mesh Reconstruction Methodscited by 3
  11. 2023PCRDiffusion: Diffusion Probabilistic Models for Point Cloud Registrationcited by 3
  12. 2024PPSURF: Combining Patches and Point Convolutions for Detailed Surface Reconstructioncited by 3
  13. 2025LayoutRAG: Retrieval-Augmented Model for Content-agnostic Conditional Layout Generationcited by 3
  14. 2025INRet: A General Framework for Accurate Retrieval of INRs for Shapescited by 0
  15. 2026FurnSet: Exploiting Repeats for 3D Scene Reconstructioncited by 0
  16. 2026CAOA -- Completion-Assisted Object-CAD Alignmentcited by 0