mlsanityData to 14 Oct 2026

Pri3D: Can 3D Priors Help 2D Representation Learning?

arXiv 2104.11225 ↗First posted 22 Apr 2021cs.CVcited by 61

Not on the map. Why?

Abstract (arXiv)

Recent advances in 3D perception have shown impressive progress in understanding geometric structures of 3Dshapes and even scenes. Inspired by these advances in geometric understanding, we aim to imbue image-based perception with representations learned under geometric constraints. We introduce an approach to learn view-invariant,geometry-aware representations for network pre-training, based on multi-view RGB-D data, that can then be effectively transferred to downstream 2D tasks. We propose to employ contrastive learning under both multi-view im-age constraints and image-geometry constraints to encode3D priors into learned 2D representations. This results not only in improvement over 2D-only representation learning on the image-based tasks of semantic segmentation, instance segmentation, and object detection on real-world in-door datasets, but moreover, provides significant improvement in the low data regime. We show a significant improvement of 6.0% on semantic segmentation on full data as well as 11.9% on 20% data against baselines on ScanNet.

Key facts. Cited by 61 papers in the map (data through 14 Oct 2026). Builds on 53 of its 67 references that are in the map. First posted on arXiv 22 Apr 2021.

Builds on

53 of 67 references.

References · 53

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  9. 2017Mask R-CNNcited by 7,277
  10. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  12. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
38 more
  1. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  2. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  3. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  4. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  5. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  6. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  7. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  8. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  9. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,205
  10. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 748
  11. 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 737
  12. 2018PointConv: Deep Convolutional Networks on 3D Point Cloudscited by 657
  13. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  14. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  15. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 504
  16. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 321
  17. 2020PointASNL: Robust Point Clouds Processing using Nonlocal Neural Networks with Adaptive Samplingcited by 239
  18. 20183D-SIS: 3D Semantic Instance Segmentation of RGB-D Scanscited by 217
  19. 2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
  20. 2020Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contextscited by 210
  21. 2020OccuSeg: Occupancy-aware 3D Instance Segmentationcited by 177
  22. 2019Learning Object Bounding Boxes for 3D Instance Segmentation on Point Cloudscited by 156
  23. 2018GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloudcited by 146
  24. 2018Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps with Accurate Object Boundariescited by 145
  25. 20203D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentationcited by 141
  26. 2020H3DNet: 3D Object Detection Using Hybrid Geometric Primitivescited by 123
  27. 2020ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votescited by 119
  28. 20183DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentationcited by 118
  29. 2016BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integrationcited by 116
  30. 2015Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Imagescited by 111
  31. 2021Bidirectional Projection Network for Cross Dimension Scene Understandingcited by 95
  32. 20193D Instance Segmentation via Multi-Task Metric Learningcited by 80
  33. 2020JSENet: Joint Semantic Segmentation and Edge Detection Network for 3D Point Cloudscited by 62
  34. 2020FPConv: Learning Local Flattening for Point Convolutioncited by 55
  35. 20213D-to-2D Distillation for Indoor Scene Parsingcited by 48
  36. 2018TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshescited by 45
  37. 2020Fusion-Aware Point Convolution for Online Semantic 3D Scene Segmentationcited by 40
  38. 2020RfD-Net: Point Scene Understanding by Semantic Instance Reconstructioncited by 36

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 19; 2023: 15; 2024: 10; 2025: 8; 2026: 7.

61 papers.

Sort:

By area:

  1. 2022CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completioncited by 160
  2. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 137
  3. 2022Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Datacited by 126
  4. 2023Segment Any Point Cloud Sequences by Distilling Vision Foundation Modelscited by 125
  5. 2024Improving 2D Feature Representations by 3D-Aware Fine-Tuningcited by 71
  6. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 70
  7. 2023Rotation-Invariant Transformer for Point Cloud Matchingcited by 69
  8. 2021SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-Training for Spatial-Aware Visual Representationscited by 55
  9. 2022Unsupervised Point Cloud Representation Learning with Deep Neural Networks: A Surveycited by 54
  10. 2023PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detectioncited by 53
  11. 2022Ponder: Point Cloud Pre-training via Neural Renderingcited by 50
  12. 2023Learning Visual Representations via Language-Guided Samplingcited by 32
38 more
  1. 2025DUNE: Distilling a Universal Encoder from Heterogeneous 2D and 3D Teacherscited by 32
  2. 2022Versatile Multi-Modal Pre-Training for Human-Centric Perceptioncited by 26
  3. 2021Multimodal Semi-Supervised Learning for 3D Objectscited by 24
  4. 2022Let Images Give You More:Point Cloud Cross-Modal Training for Shape Analysiscited by 24
  5. 2022Open-Vocabulary 3D Detection via Image-level Class and Debiased Cross-modal Contrastive Learningcited by 23
  6. 2022ImLoveNet: Misaligned Image-supported Registration Network for Low-overlap Point Cloud Pairscited by 17
  7. 2024CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusioncited by 17
  8. 20232D-3D Interlaced Transformer for Point Cloud Segmentation with Scene-Level Supervisioncited by 16
  9. 2023Mask3D: Pre-training 2D Vision Transformers by Learning Masked 3D Priorscited by 15
  10. 2023Joint Representation Learning for Text and 3D Point Cloudcited by 14
  11. 2023PointMBF: A Multi-scale Bidirectional Fusion Network for Unsupervised RGB-D Point Cloud Registrationcited by 14
  12. 2023PCR-CG: Point Cloud Registration via Deep Explicit Color and Geometrycited by 13
  13. 2024Memory-based Adapters for Online 3D Scene Perceptioncited by 13
  14. 20243D Weakly Supervised Semantic Segmentation with 2D Vision-Language Guidancecited by 12
  15. 2022PointMCD: Boosting Deep Point Cloud Encoders via Multi-view Cross-modal Distillation for 3D Shape Recognitioncited by 10
  16. 2022MvDeCor: Multi-view Dense Correspondence Learning for Fine-grained 3D Segmentationcited by 10
  17. 2024Shelf-Supervised Cross-Modal Pre-Training for 3D Object Detectioncited by 9
  18. 2026CMHANet: A Cross-Modal Hybrid Attention Network for Point Cloud Registrationcited by 9
  19. 2022Self-Supervised Learning with Multi-View Rendering for 3D Point Cloud Analysiscited by 8
  20. 2023Geometric-aware Pretraining for Vision-centric 3D Object Detectioncited by 8
  21. 2022Self-Supervised Image Representation Learning with Geometric Set Consistencycited by 7
  22. 2023Introspective Deep Metric Learningcited by 7
  23. 2022Learning Dense Object Descriptors from Multiple Views for Low-shot Category Generalizationcited by 6
  24. 2022Self-Supervised Pre-training of 3D Point Cloud Networks with Image Datacited by 5
  25. 2023Pre-Training LiDAR-Based 3D Object Detectors Through Colorizationcited by 5
  26. 2024Cross-Modal Self-Supervised Learning with Effective Contrastive Units for LiDAR Point Cloudscited by 4
  27. 2022OPERA: Omni-Supervised Representation Learning with Hierarchical Supervisionscited by 3
  28. 2024Formula-Supervised Visual-Geometric Pre-trainingcited by 3
  29. 2025Muskie: Multi-view Masked Image Modeling for 3D Vision Pre-trainingcited by 3
  30. 2022Weakly-Supervised End-to-End CAD Retrieval to Scan Objectscited by 2
  31. 2023Contrastive Learning for Self-Supervised Pre-Training of Point Cloud Segmentation Networks With Image Datacited by 2
  32. 2023M33D: Learning 3D priors using Multi-Modal Masked Autoencoders for 2D image and video understandingcited by 2
  33. 2024Enhancing Single-Slice Segmentation with 3D-to-2D Unpaired Scan Distillationcited by 2
  34. 2025Interpretable 3D Neural Object Volumes for Robust Conceptual Reasoningcited by 2
  35. 2025Cross-modal feature fusion for robust point cloud registration with ambiguous geometrycited by 2
  36. 2026CLAMP: Contrastive Learning for 3D Multi-View Action-Conditioned Robotic Manipulation Pretrainingcited by 2
  37. 2022Physically Disentangled Representationscited by 1
  38. 2022Improving Pixel-Level Contrastive Learning by Leveraging Exogenous Depth Informationcited by 1

50 most cited of 61 All 61 (JSON)