# RfD-Net: Point Scene Understanding by Semantic Instance Reconstruction

arXiv: https://arxiv.org/abs/2011.14744 · HTML: https://mlsanity.com/paper/2011.14744/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Nov 2020.

## Built on by (most cited first, 30 of 36)

- [TransformerFusion: Monocular RGB Scene Reconstruction using Transformers](https://mlsanity.com/paper/2107.02191/) (2021)
- [3D Semantic Scene Completion: a Survey](https://mlsanity.com/paper/2103.07466/) (2021)
- [Panoptic 3D Scene Reconstruction From a Single RGB Image](https://mlsanity.com/paper/2111.02444/) (2021)
- [Pri3D: Can 3D Priors Help 2D Representation Learning?](https://mlsanity.com/paper/2104.11225/) (2021)
- [NeurAR: Neural Uncertainty for Autonomous 3D Reconstruction with Implicit Neural Representations](https://mlsanity.com/paper/2207.10985/) (2022)
- [4DContrast: Contrastive Learning with Dynamic Correspondences for 3D Scene Understanding](https://mlsanity.com/paper/2112.02990/) (2021)
- [NeRF-Det: Learning Geometry-Aware Volumetric Representation for Multi-View 3D Object Detection](https://mlsanity.com/paper/2307.14620/) (2023)
- [LODE: Locally Conditioned Eikonal Implicit Scene Completion from Sparse LiDAR](https://mlsanity.com/paper/2302.14052/) (2023)
- [Learning 3D Scene Priors with 2D Supervision](https://mlsanity.com/paper/2211.14157/) (2022)
- [ODAM: Object Detection, Association, and Mapping using Posed RGB Video](https://mlsanity.com/paper/2108.10165/) (2021)
- [Pose2Room: Understanding 3D Scenes from Human Activities](https://mlsanity.com/paper/2112.03030/) (2021)
- [Point Scene Understanding via Disentangled Instance Mesh Reconstruction](https://mlsanity.com/paper/2203.16832/) (2022)
- [BUOL: A Bottom-Up Framework with Occupancy-aware Lifting for Panoptic 3D Scene Reconstruction From A Single Image](https://mlsanity.com/paper/2306.00965/) (2023)
- [EFEM: Equivariant Neural Field Expectation Maximization for 3D Object Segmentation Without Scene Supervision](https://mlsanity.com/paper/2303.15440/) (2023)
- [Mask3D: Pre-training 2D Vision Transformers by Learning Masked 3D Priors](https://mlsanity.com/paper/2302.14746/) (2023)
- [Detection Based Part-level Articulated Object Reconstruction from Single RGBD Image](https://mlsanity.com/paper/2504.03177/) (2025)
- [Semi-supervised Implicit Scene Completion from Sparse LiDAR](https://mlsanity.com/paper/2111.14798/) (2021)
- [Skip Mamba Diffusion for Monocular 3D Semantic Scene Completion](https://mlsanity.com/paper/2501.07260/) (2025)
- [Fast Adversarial Training with Weak-to-Strong Spatial-Temporal Consistency in the Frequency Domain on Videos](https://mlsanity.com/paper/2504.14921/) (2025)
- [Neural Part Priors: Learning to Optimize Part-Based Object Completion in RGB-D Scans](https://mlsanity.com/paper/2203.09375/) (2022)
- [Articulation in Motion: Prior-free Part Mobility Analysis for Articulated Objects By Dynamic-Static Disentanglement](https://mlsanity.com/paper/2603.02910/) (2026)
- [LASA: Instance Reconstruction from Real Scans using A Large-scale Aligned Shape Annotation Dataset](https://mlsanity.com/paper/2312.12418/) (2023)
- [Rethinking Multimodal Point Cloud Completion: A Completion-by-Correction Perspective](https://mlsanity.com/paper/2511.12170/) (2025)
- [UniPlane: Unified Plane Detection and Reconstruction from Posed Monocular Videos](https://mlsanity.com/paper/2407.03594/) (2024)
- [CymbaDiff: Structured Spatial Diffusion for Sketch-based 3D Semantic Urban Scene Generation](https://mlsanity.com/paper/2510.13245/) (2025)
- [ROSEFusion: Random Optimization for Online Dense Reconstruction under Fast Camera Motion](https://mlsanity.com/paper/2105.05600/) (2021)
- [Self-Ordering Point Clouds](https://mlsanity.com/paper/2304.00961/) (2023)
- [3D objects and scenes classification, recognition, segmentation, and reconstruction using 3D point cloud data: A review](https://mlsanity.com/paper/2306.05978/) (2023)
- [Shape Anchor Guided Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2309.11133/) (2023)
- [Point-based Instance Completion with Scene Constraints](https://mlsanity.com/paper/2504.05698/) (2025)

## Abstract (arXiv)

Semantic scene understanding from point clouds is particularly challenging as the points reflect only a sparse set of the underlying 3D geometry. Previous works often convert point cloud into regular grids (e.g. voxels or bird-eye view images), and resort to grid-based convolutions for scene understanding. In this work, we introduce RfD-Net that jointly detects and reconstructs dense object surfaces directly from raw point clouds. Instead of representing scenes with regular grids, our method leverages the sparsity of point cloud data and focuses on predicting shapes that are recognized with high objectness. With this design, we decouple the instance reconstruction into global object localization and local shape prediction. It not only eases the difficulty of learning 2-D manifold surfaces from sparse 3D space, the point clouds in each object proposal convey shape details that support implicit function learning to reconstruct any high-resolution surfaces. Our experiments indicate that instance detection and reconstruction present complementary effects, where the shape prediction head shows consistent effects on improving object detection with modern 3D proposal network backbones. The qualitative and quantitative evaluations further demonstrate that our approach consistently outperforms the state-of-the-arts and improves over 11 of mesh IoU in object reconstruction.