# ELiTe: Efficient Image-to-LiDAR Knowledge Transfer for Semantic Segmentation

arXiv: https://arxiv.org/abs/2405.04121 · HTML: https://mlsanity.com/paper/2405.04121/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 39 references that are in the map.
- First posted on arXiv 7 May 2024.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [PointCLIP: Point Cloud Understanding by CLIP](https://mlsanity.com/paper/2112.02413/) (2021)
- [Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution](https://mlsanity.com/paper/2007.16100/) (2020)
- [Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation](https://mlsanity.com/paper/2011.10033/) (2020)
- [PolarNet: An Improved Grid Representation for Online LiDAR Point Clouds Semantic Segmentation](https://mlsanity.com/paper/2003.14032/) (2020)
- [RPVNet: A Deep and Efficient Range-Point-Voxel Fusion Network for LiDAR Point Cloud Segmentation](https://mlsanity.com/paper/2103.12978/) (2021)
- [(AF)2-S3Net: Attentive Feature Fusion with Adaptive Feature Selection for Sparse Semantic Segmentation Network](https://mlsanity.com/paper/2102.04530/) (2021)
- [2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds](https://mlsanity.com/paper/2207.04397/) (2022)
- [SAM3D: Segment Anything in 3D Scenes](https://mlsanity.com/paper/2306.03908/) (2023)
- [Point-to-Voxel Knowledge Distillation for LiDAR Semantic Segmentation](https://mlsanity.com/paper/2206.02099/) (2022)
- [On the Role of Neural Collapse in Transfer Learning](https://mlsanity.com/paper/2112.15121/) (2021)
- [Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Models](https://mlsanity.com/paper/2106.04180/) (2021)
- [SAM3D: Zero-Shot 3D Object Detection via Segment Anything Model](https://mlsanity.com/paper/2306.02245/) (2023)
- [RGB and LiDAR fusion based 3D Semantic Segmentation for Autonomous Driving](https://mlsanity.com/paper/1906.00208/) (2019)
- [Lite-HDSeg: LiDAR Semantic Segmentation Using Lite Harmonic Dense Convolutions](https://mlsanity.com/paper/2103.08852/) (2021)
- [TORNADO-Net: mulTiview tOtal vaRiatioN semAntic segmentation with Diamond inceptiOn module](https://mlsanity.com/paper/2008.10544/) (2020)
- [Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuning](https://mlsanity.com/paper/2305.15212/) (2023)
- [Attention-based Transformation from Latent Features to Point Clouds](https://mlsanity.com/paper/2112.05324/) (2021)
- [Generate Point Clouds with Multiscale Details from Graph-Represented Structures](https://mlsanity.com/paper/2112.06433/) (2021)
- [Robust Fine-tuning for Pre-trained 3D Point Cloud Models](https://mlsanity.com/paper/2404.16422/) (2024)

## Abstract (arXiv)

Cross-modal knowledge transfer enhances point cloud representation learning in LiDAR semantic segmentation. Despite its potential, the textitweak teacher challenge arises due to repetitive and non-diverse car camera images and sparse, inaccurate ground truth labels. To address this, we propose the Efficient Image-to-LiDAR Knowledge Transfer (ELiTe) paradigm. ELiTe introduces Patch-to-Point Multi-Stage Knowledge Distillation, transferring comprehensive knowledge from the Vision Foundation Model (VFM), extensively trained on diverse open-world images. This enables effective knowledge transfer to a lightweight student model across modalities. ELiTe employs Parameter-Efficient Fine-Tuning to strengthen the VFM teacher and expedite large-scale model training with minimal costs. Additionally, we introduce the Segment Anything Model based Pseudo-Label Generation approach to enhance low-quality image labels, facilitating robust semantic representations. Efficient knowledge transfer in ELiTe yields state-of-the-art results on the SemanticKITTI benchmark, outperforming real-time inference models. Our approach achieves this with significantly fewer parameters, confirming its effectiveness and efficiency.