mlsanityData to 7 Oct 2026

What's Behind the Couch? Directed Ray Distance Functions (DRDF) for 3D Scene Reconstruction

arXiv 2112.04481 ↗First posted 8 Dec 2021cs.CV cs.GRcited by 7

Not on the map. Why?

Abstract (arXiv)

We present an approach for full 3D scene reconstruction from a single unseen image. We train on dataset of realistic non-watertight scans of scenes. Our approach predicts a distance function, since these have shown promise in handling complex topologies and large spaces. We identify and analyze two key challenges for predicting such image conditioned distance functions that have prevented their success on real 3D scene data. First, we show that predicting a conventional scene distance from an image requires reasoning over a large receptive field. Second, we analytically show that the optimal output of the network trained to predict these distance functions does not obey all the distance function properties. We propose an alternate distance function, the Directed Ray Distance Function (DRDF), that tackles both challenges. We show that a deep network trained to predict DRDFs outperforms all other methods quantitatively and qualitatively on 3D reconstruction from single image on Matterport3D, 3DFront, and ScanNet.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 55 of its 73 references that are in the map. First posted on arXiv 8 Dec 2021.

Builds on

55 of 73 references.

References · 55

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  4. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  5. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  6. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  7. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  8. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  9. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  10. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  11. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  12. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
38 more
  1. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  2. 2019Objects as Pointscited by 1,172
  3. 2016Colorful Image Colorizationcited by 1,113
  4. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  5. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  6. 2020NeRF++: Analyzing and Improving Neural Radiance Fieldscited by 857
  7. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
  8. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 649
  9. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 587
  10. 2018AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generationcited by 539
  11. 2020PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitizationcited by 521
  12. 2016Semantic Scene Completion from a Single Depth Imagecited by 468
  13. 2019SAL: Sign Agnostic Learning of Shapes from Raw Datacited by 389
  14. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 375
  15. 2019DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstructioncited by 332
  16. 20163D Bounding Box Estimation Using Deep Learning and Geometrycited by 277
  17. 2020Neural Unsigned Distance Fields for Implicit Function Learningcited by 265
  18. 2019What Do Single-view 3D Reconstruction Networks Learn?cited by 259
  19. 2021NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Videocited by 241
  20. 2018Pix3D: Dataset and Methods for Single-Image 3D Shape Modelingcited by 218
  21. 2020On the uncertainty of self-supervised monocular depth estimationcited by 182
  22. 2016Learning Shape Abstractions by Assembling Volumetric Primitivescited by 181
  23. 2016Learning a Predictable and Generative Vector Representation for Objectscited by 173
  24. 2020FCOS: A simple and strong anchor-free object detectorcited by 171
  25. 2020Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Imagecited by 170
  26. 2017Learning Efficient Point Cloud Generation for Dense 3D Object Reconstructioncited by 133
  27. 2021Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimationcited by 116
  28. 2020SALD: Sign Agnostic Learning with Derivativescited by 115
  29. 2019Monocular 3D Object Detection Leveraging Accurate Proposals and Shape Reconstructioncited by 106
  30. 2020Improving data-driven global weather prediction using deep convolutional neural networks on a cubed spherecited by 95
  31. 2017Hierarchical Surface Prediction for 3D Object Reconstructioncited by 89
  32. 2018Layer-structured 3D Scene Inference via View Synthesiscited by 82
  33. 2016IM2CADcited by 78
  34. 2019SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scanscited by 77
  35. 2019Canonical Surface Mapping via Geometric Cycle Consistencycited by 62
  36. 2014Designing Deep Networks for Surface Normal Estimationcited by 59
  37. 2017Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scenecited by 50
  38. 2019SciPy 1.0--Fundamental Algorithms for Scientific Computing in Pythoncited by 49

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 1; 2023: 2; 2024: 1; 2025: 1; 2026: 1.

7 papers.

  1. 2025SAM 3D: 3Dfy Anything in Imagescited by 230
  2. 2023Multiview Compressive Coding for 3D Reconstructioncited by 71
  3. 2021Representing 3D Shapes with Probabilistic Directed Distance Fieldscited by 21
  4. 2022Neural Correspondence Field for Object Pose Estimationcited by 21
  5. 2023Neural Voting Field for Camera-Space 3D Hand Pose Estimationcited by 6
  6. 2024IPoD: Implicit Field Learning with Point Diffusion for Generalizable 3D Object Reconstruction from Single RGB-D Imagescited by 1
  7. 2026CAOA -- Completion-Assisted Object-CAD Alignmentcited by 0