mlsanityData to 14 Oct 2026

Contrastive Learning of Features between Images and LiDAR

arXiv 2206.12071 ↗First posted 24 Jun 2022cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Image and Point Clouds provide different information for robots. Finding the correspondences between data from different sensors is crucial for various tasks such as localization, mapping, and navigation. Learning-based descriptors have been developed for single sensors; there is little work on cross-modal features. This work treats learning cross-modal features as a dense contrastive learning problem. We propose a Tuple-Circle loss function for cross-modality feature learning. Furthermore, to learn good features and not lose generality, we developed a variant of widely used PointNet++ architecture for point cloud and U-Net CNN architecture for images. Moreover, we conduct experiments on a real-world dataset to show the effectiveness of our loss function and network structure. We show that our models indeed learn information from both images as well as LiDAR by visualizing the features.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 21 of its 34 references that are in the map. First posted on arXiv 24 Jun 2022.

Builds on

21 of 34 references.

References · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  5. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
  6. 2017Deformable Convolutional Networkscited by 1,488
  7. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  8. 2016Geometric deep learning: going beyond Euclidean datacited by 1,048
  9. 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 575
  10. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  11. 2020Circle Loss: A Unified Perspective of Pair Similarity Optimizationcited by 403
  12. 2020Keystroke Dynamics as Part of Lifeloggingcited by 334
9 more
  1. 2016Domain Separation Networkscited by 320
  2. 2020RELLIS-3D Dataset: Data, Benchmarks and Analysiscited by 119
  3. 20192D3D-MatchNet: Learning to Match Keypoints Across 2D Image and 3D Point Cloudcited by 62
  4. 2021Contrastive Multimodal Fusion with TupleInfoNCEcited by 43
  5. 2021P2-Net: Joint Description and Detection of Local Features for Pixel and Point Matchingcited by 42
  6. 2019LCD: Learned Cross-Domain Descriptors for 2D-3D Matchingcited by 29
  7. 2019Global visual localization in LiDAR-maps through shared 2D-3D embedding spacecited by 28
  8. 2021i3dLoc: Image-to-range Cross-domain Localization Robust to Inconsistent Environmental Conditionscited by 14
  9. 2021SemCal: Semantic LiDAR-Camera Calibration using Neural MutualInformation Estimatorcited by 7

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 1; 2025: 1.

5 papers.

  1. 2023FreeReg: Image-to-Point Cloud Registration Leveraging Pretrained Diffusion Models and Monocular Depth Estimatorscited by 31
  2. 20243DGS-ReLoc: 3D Gaussian Splatting for Map Representation and Visual ReLocalizationcited by 15
  3. 2023ViPFormer: Efficient Vision-and-Pointcloud Transformer for Unsupervised Pointcloud Understandingcited by 8
  4. 2025Diff2I2P: Differentiable Image-to-Point Cloud Registration with Diffusion Priorcited by 8
  5. 2023Improving Extrinsics between RADAR and LIDAR using Learningcited by 2