# Geometry-Guided Semantic Reconstruction for 3D Instance Segmentation

arXiv: https://arxiv.org/abs/2610.16122 · HTML: https://mlsanity.com/paper/2610.16122/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 39 of its 48 references that are in the map.
- First posted on arXiv 12 Oct 2026.

## Builds on (most cited first, 30 of 39)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling](https://mlsanity.com/paper/2111.14819/) (2021)
- [Point Transformer V3: Simpler, Faster, Stronger](https://mlsanity.com/paper/2312.10035/) (2023)
- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [Deep Learning for 3D Point Clouds: A Survey](https://mlsanity.com/paper/1912.12033/) (2019)
- [Large-scale Point Cloud Semantic Segmentation with Superpoint Graphs](https://mlsanity.com/paper/1711.09869/) (2017)
- [A Survey of Autonomous Driving: Common Practices and Emerging Technologies](https://mlsanity.com/paper/1906.05113/) (2019)
- [PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation](https://mlsanity.com/paper/2004.01658/) (2020)
- [Mask3D: Mask Transformer for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2210.03105/) (2022)
- [Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training](https://mlsanity.com/paper/2205.14401/) (2022)
- [Language-Grounded Indoor 3D Semantic Segmentation in the Wild](https://mlsanity.com/paper/2204.07761/) (2022)
- [SoftGroup for 3D Instance Segmentation on Point Clouds](https://mlsanity.com/paper/2203.01509/) (2022)
- [CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIP](https://mlsanity.com/paper/2301.04926/) (2023)
- [Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining](https://mlsanity.com/paper/2302.02318/) (2023)
- [Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoders](https://mlsanity.com/paper/2212.06785/) (2022)
- [3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2003.13867/) (2020)
- [Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance](https://mlsanity.com/paper/2312.10671/) (2023)
- [Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?](https://mlsanity.com/paper/2212.08320/) (2022)
- [Superpoint Transformer for 3D Scene Instance Segmentation](https://mlsanity.com/paper/2211.15766/) (2022)
- [Instance Segmentation in 3D Scenes using Semantic Superpoint Tree Networks](https://mlsanity.com/paper/2108.07478/) (2021)
- [DINO-X: A Unified Vision Model for Open-World Object Detection and Understanding](https://mlsanity.com/paper/2411.14347/) (2024)
- [3D Instance Segmentation via Multi-Task Metric Learning](https://mlsanity.com/paper/1906.08650/) (2019)
- [DyCo3D: Robust Instance Segmentation of 3D Point Clouds through Dynamic Convolution](https://mlsanity.com/paper/2011.13328/) (2020)
- [ISBNet: a 3D Point Cloud Instance Segmentation Network with Instance-aware Sampling and Box-aware Dynamic Convolution](https://mlsanity.com/paper/2303.00246/) (2023)
- [Mask-Attention-Free Transformer for 3D Instance Segmentation](https://mlsanity.com/paper/2309.01692/) (2023)
- [MaskGroup: Hierarchical Point Grouping and Masking for 3D Instance Segmentation](https://mlsanity.com/paper/2203.14662/) (2022)

## Abstract (arXiv)

3D instance segmentation requires representations that are both geometrically structured and semantically discriminative. Pure 3D encoders model spatial structure well but often lack strong appearance semantics, while directly lifting 2D priors into 3D by concatenation injects rich semantics without ensuring that they align well with the underlying 3D structure. In this paper, we address this problem by formulating 2D-to-3D semantic transfer as a masked semantic reconstruction task. Specifically, we mask a subset of background lifted 2D priors and train the 3D encoder to recover the missing semantic priors, providing explicit supervision for distilling rich 2D semantics into the 3D backbone. We further propose geometry-modulated semantic injection, which uses raw 3D geometry to modulate lifted 2D priors before sparse encoding, so that geometry explicitly controls semantic injection and alleviates the distribution gap between 2D semantic features and 3D geometric features. Experiments on two baseline frameworks demonstrate that the proposed method achieves state-of-the-art results in both full-data and few-shot settings on ScanNetV2 and ScanNet200. Code is publicly available at https://github.com/yharoldsu0627/GeometryGuidedSemanticReconstruction