mlsanityData to 7 Oct 2026

3D-to-2D Distillation for Indoor Scene Parsing

arXiv 2104.02243 ↗First posted 6 Apr 2021cs.CVcited by 47

Not on the map. Why?

Abstract (arXiv)

Indoor scene semantic parsing from RGB images is very challenging due to occlusions, object distortion, and viewpoint variations. Going beyond prior works that leverage geometry information, typically paired depth maps, we present a new approach, a 3D-to-2D distillation framework, that enables us to leverage 3D features extracted from large-scale 3D data repository (e.g., ScanNet-v2) to enhance 2D features extracted from RGB images. Our work has three novel contributions. First, we distill 3D knowledge from a pretrained 3D network to supervise a 2D network to learn simulated 3D features from 2D features during the training, so the 2D network can infer without requiring 3D data. Second, we design a two-stage dimension normalization scheme to calibrate the 2D and 3D features for better integration. Third, we design a semantic-aware adversarial training model to extend our framework for training with unpaired 3D data. Extensive experiments on various datasets, ScanNet-V2, S3DIS, and NYU-v2, demonstrate the superiority of our approach. Also, experimental results show that our 3D-to-2D distillation improves the model generalization.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 49 of its 67 references that are in the map. First posted on arXiv 6 Apr 2021.

Builds on

49 of 67 references.

References · 49

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  6. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  7. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  8. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  9. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  10. 2016Pyramid Scene Parsing Networkcited by 2,472
  11. 2017Non-local Neural Networkscited by 2,290
  12. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
37 more
  1. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  2. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,483
  3. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,417
  4. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,202
  5. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  6. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,128
  7. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  8. 2018PointConv: Deep Convolutional Networks on 3D Point Cloudscited by 656
  9. 2019Segmentation Transformer: Object-Contextual Representations for Semantic Segmentationcited by 573
  10. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 515
  11. 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
  12. 2017Joint 2D-3D-Semantic Data for Indoor Scene Understandingcited by 404
  13. 2019From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimationcited by 379
  14. 2018How Does Batch Normalization Help Optimization?cited by 355
  15. 2016OctNet: Learning Deep 3D Representations at High Resolutionscited by 340
  16. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 318
  17. 2016ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentationcited by 291
  18. 2016Revisiting Batch Normalization For Practical Domain Adaptationcited by 285
  19. 2017ICNet for Real-Time Semantic Segmentation on High-Resolution Imagescited by 261
  20. 2018PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsingcited by 224
  21. 2020OccuSeg: Occupancy-aware 3D Instance Segmentationcited by 177
  22. 2015ParseNet: Looking Wider to See Bettercited by 176
  23. 2015Cross Modal Distillation for Supervision Transfercited by 173
  24. 2019Pattern-Affinitive Propagation across Depth, Surface Normal and Semantic Segmentationcited by 141
  25. 2018OCNet: Object Context Network for Scene Parsingcited by 140
  26. 2014Learning Rich Features from RGB-D Images for Object Detection and Segmentationcited by 124
  27. 20183DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentationcited by 118
  28. 2020Virtual Multi-view Fusion for 3D Semantic Segmentationcited by 99
  29. 2018Modality Distillation with Multiple Stream Networks for Action Recognitioncited by 83
  30. 2018Self-Supervised Model Adaptation for Multimodal Semantic Segmentationcited by 73
  31. 2015Sparse 3D convolutional neural networkscited by 71
  32. 2019ConvPoint: Continuous Convolutions for Point Cloud Processingcited by 62
  33. 2018Knowledge Distillation with Adversarial Samples Supporting Decision Boundarycited by 49
  34. 2018Learning with privileged information via adversarial discriminative modality distillationcited by 30
  35. 2019RFBNet: Deep Multimodal Networks with Residual Fusion Blocks for RGB-D Semantic Segmentationcited by 12
  36. 2018Cross-Modality Distillation: A case for Conditional Generative Adversarial Networkscited by 11
  37. 2018Modeling 4D fMRI Data via Spatio-Temporal Convolutional Neural Networks (ST-CNN)cited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 18; 2023: 8; 2024: 8; 2025: 6; 2026: 3.

47 papers.

Sort:

By area:

  1. 2024DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videoscited by 202
  2. 20222DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Cloudscited by 157
  3. 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 143
  4. 2022Neural 3D Scene Reconstruction with the Manhattan-world Assumptioncited by 123
  5. 2022UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Groundingcited by 69
  6. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 63
  7. 2021Pri3D: Can 3D Priors Help 2D Representation Learning?cited by 61
  8. 2021Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Modelscited by 41
  9. 2023EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understandingcited by 32
  10. 2022Towards Efficient 3D Object Detection with Knowledge Distillationcited by 29
  11. 2023X3KD: Knowledge Distillation Across Modalities, Tasks and Stages for Multi-Camera 3D Object Detectioncited by 28
  12. 2022A Closer Look at Invariances in Self-supervised Pre-training for 3D Visioncited by 24
35 more
  1. 2022Let Images Give You More:Point Cloud Cross-Modal Training for Shape Analysiscited by 24
  2. 2021Voint Cloud: Multi-View Point Cloud Representation for 3D Understandingcited by 22
  3. 2022Boosting 3D Object Detection by Simulating Multimodality on Point Cloudscited by 22
  4. 2022DetMatch: Two Teachers are Better Than One for Joint 2D and 3D Semi-Supervised Object Detectioncited by 20
  5. 2022ImLoveNet: Misaligned Image-supported Registration Network for Low-overlap Point Cloud Pairscited by 17
  6. 2023Panoptic Vision-Language Feature Fieldscited by 17
  7. 2023PCR-CG: Point Cloud Registration via Deep Explicit Color and Geometrycited by 13
  8. 2024Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Modelscited by 12
  9. 2024HVDistill: Transferring Knowledge from Images to Point Clouds via Unsupervised Hybrid-View Distillationcited by 11
  10. 2022PointMCD: Boosting Deep Point Cloud Encoders via Multi-view Cross-modal Distillation for 3D Shape Recognitioncited by 10
  11. 2023Mx2M: Masked Cross-Modality Modeling in Domain Adaptation for 3D Semantic Segmentationcited by 10
  12. 2021HIDA: Towards Holistic Indoor Understanding for the Visually Impaired via Semantic Instance Segmentation with a Wearable Solid-State LiDAR Sensorcited by 9
  13. 2022Self-Supervised Learning with Multi-View Rendering for 3D Point Cloud Analysiscited by 8
  14. 2022Self-Supervised Image Representation Learning with Geometric Set Consistencycited by 7
  15. 2023Transferring CLIP's Knowledge into Zero-Shot Point Cloud Semantic Segmentationcited by 7
  16. 2022Pix4Point: Image Pretrained Standard Transformers for 3D Point Cloud Understandingcited by 6
  17. 2022Language-Assisted 3D Feature Learning for Semantic Scene Understandingcited by 6
  18. 2024Make a Strong Teacher with Label Assistance: A Novel Knowledge Distillation Approach for Semantic Segmentationcited by 5
  19. 2023Cross-modal & Cross-domain Learning for Unsupervised LiDAR Semantic Segmentationcited by 4
  20. 2025EgoAdapt: Adaptive Multisensory Distillation and Policy Learning for Efficient Egocentric Perceptioncited by 4
  21. 2024Formula-Supervised Visual-Geometric Pre-trainingcited by 3
  22. 2022Hilbert Distillation for Cross-Dimensionality Networkscited by 2
  23. 2024Enhancing Single-Slice Segmentation with 3D-to-2D Unpaired Scan Distillationcited by 2
  24. 2022Lightweight integration of 3D features to improve 2D image segmentationcited by 1
  25. 20252D-3D Attention and Entropy for Pose Robust 2D Facial Recognitioncited by 1
  26. 2022Offset-Guided Attention Network for Room-Level Aware Floor Plan Segmentationcited by 0
  27. 2023360^circ High-Resolution Depth Estimation via Uncertainty-aware Structural Knowledge Transfercited by 0
  28. 2024Boosting Cross-Domain Point Classification via Distilling Relational Priors from 2D Transformerscited by 0
  29. 20243DPX: Single Panoramic X-ray Analysis Guided by 3D Oral Structure Reconstructioncited by 0
  30. 2025Project-and-Fuse: Improving RGB-D Semantic Segmentation via Graph Convolution Networkscited by 0
  31. 2025GAA-TSO: Geometry-Aware Assisted Depth Completion for Transparent and Specular Objectscited by 0
  32. 2025CasPoinTr: Point Cloud Completion with Cascaded Networks and Knowledge Distillationcited by 0
  33. 20263D Reconstruction and Knowledge Distillation to Improve Multi-View Image Models to Explore Spike Volume Estimation in Wheatcited by 0
  34. 2026Boosting Text-Driven Video Segmentation via Geometry-Aware Distillationcited by 0
  35. 2026Distill What RGB Can Recover: Privileged 3D Evidence for RGB-Only Vision-Language Modelscited by 0