# Point Scene Understanding via Disentangled Instance Mesh Reconstruction

arXiv: https://arxiv.org/abs/2203.16832 · HTML: https://mlsanity.com/paper/2203.16832/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 74 references that are in the map.
- First posted on arXiv 31 Mar 2022.

## Builds on (most cited first, 30 of 64)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling](https://mlsanity.com/paper/1610.07584/) (2016)
- [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016)
- [Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution](https://mlsanity.com/paper/2007.16100/) (2020)
- [PCN: Point Completion Network](https://mlsanity.com/paper/1808.00671/) (2018)
- [Point-Voxel CNN for Efficient 3D Deep Learning](https://mlsanity.com/paper/1907.03739/) (2019)
- [Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion](https://mlsanity.com/paper/2003.01456/) (2020)
- [PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation](https://mlsanity.com/paper/2004.01658/) (2020)
- [Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1706.01307/) (2017)
- [Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis](https://mlsanity.com/paper/1612.00101/) (2016)
- [3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans](https://mlsanity.com/paper/1812.07003/) (2018)
- [Group-Free 3D Object Detection via Transformers](https://mlsanity.com/paper/2104.00678/) (2021)
- [BSP-Net: Generating Compact Meshes via Binary Space Partitioning](https://mlsanity.com/paper/1911.06971/) (2019)
- [OccuSeg: Occupancy-aware 3D Instance Segmentation](https://mlsanity.com/paper/2003.06537/) (2020)
- [Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image](https://mlsanity.com/paper/2002.12212/) (2020)
- [SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation](https://mlsanity.com/paper/1711.08588/) (2017)
- [Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds](https://mlsanity.com/paper/1906.01140/) (2019)
- [Scan2CAD: Learning CAD Model Alignment in RGB-D Scans](https://mlsanity.com/paper/1811.11187/) (2018)
- [3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2003.13867/) (2020)
- [Hierarchical Aggregation for 3D Instance Segmentation](https://mlsanity.com/paper/2108.02350/) (2021)
- [3D Sketch-aware Semantic Scene Completion via Semi-supervised Structure Prior](https://mlsanity.com/paper/2003.14052/) (2020)
- [PQ-NET: A Generative Part Seq2Seq Network for 3D Shapes](https://mlsanity.com/paper/1911.10949/) (2019)
- [Cascaded Refinement Network for Point Cloud Completion](https://mlsanity.com/paper/2004.03327/) (2020)

## Built on by (most cited first, 20 of 20)

- [Compressible-composable NeRF via Rank-residual Decomposition](https://mlsanity.com/paper/2205.14870/) (2022)
- [Uncovering and Quantifying Social Biases in Code Generation](https://mlsanity.com/paper/2305.15377/) (2023)
- [Interactive Segment Anything NeRF with Feature Imitation](https://mlsanity.com/paper/2305.16233/) (2023)
- [Conditional DETR V2: Efficient Detection Transformer with Box Queries](https://mlsanity.com/paper/2207.08914/) (2022)
- [D-TensoRF: Tensorial Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2212.02375/) (2022)
- [ChatCam: Empowering Camera Control through Conversational AI](https://mlsanity.com/paper/2409.17331/) (2024)
- [Detection Based Part-level Articulated Object Reconstruction from Single RGBD Image](https://mlsanity.com/paper/2504.03177/) (2025)
- [3D Question Answering for City Scene Understanding](https://mlsanity.com/paper/2407.17398/) (2024)
- [Spectrum AUC Difference (SAUCD): Human-aligned 3D Shape Evaluation](https://mlsanity.com/paper/2403.01619/) (2024)
- [Gear-NeRF: Free-Viewpoint Rendering and Tracking with Motion-aware Spatio-Temporal Sampling](https://mlsanity.com/paper/2406.03723/) (2024)
- [Uncovering and Categorizing Social Biases in Text-to-SQL](https://mlsanity.com/paper/2305.16253/) (2023)
- [LASA: Instance Reconstruction from Real Scans using A Large-scale Aligned Shape Annotation Dataset](https://mlsanity.com/paper/2312.12418/) (2023)
- [Parallel Sentence-Level Explanation Generation for Real-World Low-Resource Scenarios](https://mlsanity.com/paper/2302.10707/) (2023)
- [Real-time 3D Semantic Scene Completion Via Feature Aggregation and Conditioned Prediction](https://mlsanity.com/paper/2303.10967/) (2023)
- [Improving Long Text Understanding with Knowledge Distilled from Summarization Model](https://mlsanity.com/paper/2405.04955/) (2024)
- [Shape Anchor Guided Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2309.11133/) (2023)
- [Point-based Instance Completion with Scene Constraints](https://mlsanity.com/paper/2504.05698/) (2025)
- [3D Point World Models: Point Completion Enables More Accurate Dynamics Learning](https://mlsanity.com/paper/2607.00148/) (2026)
- [DOCTR: Disentangled Object-Centric Transformer for Point Scene Understanding](https://mlsanity.com/paper/2403.16431/) (2024)
- [SRAM: Shape-Realism Alignment Metric for No Reference 3D Shape Evaluation](https://mlsanity.com/paper/2512.01373/) (2025)

## Abstract (arXiv)

Semantic scene reconstruction from point cloud is an essential and challenging task for 3D scene understanding. This task requires not only to recognize each instance in the scene, but also to recover their geometries based on the partial observed point cloud. Existing methods usually attempt to directly predict occupancy values of the complete object based on incomplete point cloud proposals from a detection-based backbone. However, this framework always fails to reconstruct high fidelity mesh due to the obstruction of various detected false positive object proposals and the ambiguity of incomplete point observations for learning occupancy values of complete objects. To circumvent the hurdle, we propose a Disentangled Instance Mesh Reconstruction (DIMR) framework for effective point scene understanding. A segmentation-based backbone is applied to reduce false positive object proposals, which further benefits our exploration on the relationship between recognition and reconstruction. Based on the accurate proposals, we leverage a mesh-aware latent code space to disentangle the processes of shape completion and mesh generation, relieving the ambiguity caused by the incomplete point observations. Furthermore, with access to the CAD model pool at test time, our model can also be used to improve the reconstruction quality by performing mesh retrieval without extra training. We thoroughly evaluate the reconstructed mesh quality with multiple metrics, and demonstrate the superiority of our method on the challenging ScanNet dataset. Code is available at urlhttps://github.com/ashawkey/dimr.