# Self-Supervised Image-to-Point Distillation via Semantically Tolerant Contrastive Loss

arXiv: https://arxiv.org/abs/2301.05709 · HTML: https://mlsanity.com/paper/2301.05709/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 31 references that are in the map.
- First posted on arXiv 12 Jan 2023.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [Dense Contrastive Learning for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2011.09157/) (2020)
- [PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding](https://mlsanity.com/paper/2007.10985/) (2020)
- [Understanding the Behaviour of Contrastive Loss](https://mlsanity.com/paper/2012.09740/) (2020)
- [Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation](https://mlsanity.com/paper/2011.10033/) (2020)
- [Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection](https://mlsanity.com/paper/1908.09492/) (2019)
- [Self-Supervised Pretraining of 3D Features on any Point-Cloud](https://mlsanity.com/paper/2101.02691/) (2021)
- [Efficient Visual Pretraining with Contrastive Detection](https://mlsanity.com/paper/2103.10957/) (2021)
- [Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Data](https://mlsanity.com/paper/2203.16258/) (2022)
- [Guided Point Contrastive Learning for Semi-supervised Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2110.08188/) (2021)
- [Point Density-Aware Voxels for LiDAR 3D Object Detection](https://mlsanity.com/paper/2203.05662/) (2022)
- [OBoW: Online Bag-of-Visual-Words Generation for Self-Supervised Learning](https://mlsanity.com/paper/2012.11552/) (2020)
- [Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretraining](https://mlsanity.com/paper/2104.04687/) (2021)
- [A Closer Look at Invariances in Self-supervised Pre-training for 3D Vision](https://mlsanity.com/paper/2207.04997/) (2022)
- [Dense Voxel Fusion for 3D Object Detection](https://mlsanity.com/paper/2203.00871/) (2022)

## Built on by (most cited first, 30 of 35)

- [Segment Any Point Cloud Sequences by Distilling Vision Foundation Models](https://mlsanity.com/paper/2306.09347/) (2023)
- [Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Driving](https://mlsanity.com/paper/2405.05258/) (2024)
- [A Survey for Foundation Models in Autonomous Driving](https://mlsanity.com/paper/2402.01105/) (2024)
- [Three Pillars improving Vision Foundation Model Distillation for Lidar](https://mlsanity.com/paper/2310.17504/) (2023)
- [4D Contrastive Superflows are Dense 3D Representation Learners](https://mlsanity.com/paper/2407.06190/) (2024)
- [DINO in the Room: Leveraging 2D Foundation Models for 3D Segmentation](https://mlsanity.com/paper/2503.18944/) (2025)
- [PartDistill: 3D Shape Part Segmentation by Vision-Language Model Distillation](https://mlsanity.com/paper/2312.04016/) (2023)
- [BEVContrast: Self-Supervision in BEV Space for Automotive Lidar Point Clouds](https://mlsanity.com/paper/2310.17281/) (2023)
- [Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunities](https://mlsanity.com/paper/2401.08045/) (2024)
- [LiMoE: Mixture of LiDAR Representation Learners from Automotive Scenes](https://mlsanity.com/paper/2501.04004/) (2025)
- [OccFeat: Self-supervised Occupancy Feature Prediction for Pretraining BEV Segmentation Networks](https://mlsanity.com/paper/2404.14027/) (2024)
- [360^circ from a Single Camera: A Few-Shot Approach for LiDAR Segmentation](https://mlsanity.com/paper/2309.06197/) (2023)
- [LargeAD: Large-Scale Cross-Sensor Data Pretraining for Autonomous Driving](https://mlsanity.com/paper/2501.04005/) (2025)
- [3EED: Ground Everything Everywhere in 3D](https://mlsanity.com/paper/2511.01755/) (2025)
- [Shelf-Supervised Cross-Modal Pre-Training for 3D Object Detection](https://mlsanity.com/paper/2406.10115/) (2024)
- [Label-Efficient LiDAR Semantic Segmentation with 2D-3D Vision Transformer Adapters](https://mlsanity.com/paper/2503.03299/) (2025)
- [Building a Strong Pre-Training Baseline for Universal 3D Large-Scale Perception](https://mlsanity.com/paper/2405.07201/) (2024)
- [LeAP: Consistent multi-domain 3D labeling using Foundation Models](https://mlsanity.com/paper/2502.03901/) (2025)
- [Self-supervised Learning of LiDAR 3D Point Clouds via 2D-3D Neural Calibration](https://mlsanity.com/paper/2401.12452/) (2024)
- [Beyond One Shot, Beyond One Perspective: Cross-View and Long-Horizon Distillation for Better LiDAR Representations](https://mlsanity.com/paper/2507.05260/) (2025)
- [TTT-KD: Test-Time Training for 3D Semantic Segmentation through Knowledge Distillation from Foundation Models](https://mlsanity.com/paper/2403.11691/) (2024)
- [Fine-grained Image-to-LiDAR Contrastive Distillation with Visual Foundation Models](https://mlsanity.com/paper/2405.14271/) (2024)
- [3D Annotation-Free Learning by Distilling 2D Open-Vocabulary Segmentation Models for Autonomous Driving](https://mlsanity.com/paper/2405.15286/) (2024)
- [Enhanced Spatiotemporal Consistency for Image-to-LiDAR Data Pretraining](https://mlsanity.com/paper/2503.19912/) (2025)
- [Forging Spatial Intelligence: A Roadmap of Multi-Modal Data Pre-Training for Autonomous Systems](https://mlsanity.com/paper/2512.24385/) (2025)
- [Model2Scene: Learning 3D Scene Representation via Contrastive Language-CAD Models Pre-training](https://mlsanity.com/paper/2309.16956/) (2023)
- [Image-to-Lidar Relational Distillation for Autonomous Driving Data](https://mlsanity.com/paper/2409.00845/) (2024)
- [Is Contrastive Distillation Enough for Learning Comprehensive 3D Representations?](https://mlsanity.com/paper/2412.08973/) (2024)
- [PSA-SSL: Pose and Size-aware Self-Supervised Learning on LiDAR Point Clouds](https://mlsanity.com/paper/2503.13914/) (2025)
- [Improving Multimodal Distillation for 3D Semantic Segmentation under Domain Shift](https://mlsanity.com/paper/2511.17455/) (2025)

## Abstract (arXiv)

An effective framework for learning 3D representations for perception tasks is distilling rich self-supervised image features via contrastive learning. However, image-to point representation learning for autonomous driving datasets faces two main challenges: 1) the abundance of self-similarity, which results in the contrastive losses pushing away semantically similar point and image regions and thus disturbing the local semantic structure of the learned representations, and 2) severe class imbalance as pretraining gets dominated by over-represented classes. We propose to alleviate the self-similarity problem through a novel semantically tolerant image-to-point contrastive loss that takes into consideration the semantic distance between positive and negative image regions to minimize contrasting semantically similar point and image regions. Additionally, we address class imbalance by designing a class-agnostic balanced loss that approximates the degree of class imbalance through an aggregate sample-to-samples semantic similarity measure. We demonstrate that our semantically-tolerant contrastive loss with class balancing improves state-of-the art 2D-to-3D representation learning in all evaluation settings on 3D semantic segmentation. Our method consistently outperforms state-of-the-art 2D-to-3D representation learning frameworks across a wide range of 2D self-supervised pretrained models.