# Self-supervised Feature Learning by Cross-modality and Cross-view Correspondences

arXiv: https://arxiv.org/abs/2004.05749 · HTML: https://mlsanity.com/paper/2004.05749/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Apr 2020.

## Built on by (most cited first, 19 of 19)

- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Data](https://mlsanity.com/paper/2203.16258/) (2022)
- [Self-Supervised Multimodal Learning: A Survey](https://mlsanity.com/paper/2304.01008/) (2023)
- [Unsupervised Point Cloud Representation Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2202.13589/) (2022)
- [Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Data](https://mlsanity.com/paper/2206.02353/) (2022)
- [Multimodal Semi-Supervised Learning for 3D Objects](https://mlsanity.com/paper/2110.11601/) (2021)
- [Self-supervised Learning for Pre-Training 3D Point Clouds: A Survey](https://mlsanity.com/paper/2305.04691/) (2023)
- [POS-BERT: Point Cloud One-Stage BERT Pre-Training](https://mlsanity.com/paper/2204.00989/) (2022)
- [Mx2M: Masked Cross-Modality Modeling in Domain Adaptation for 3D Semantic Segmentation](https://mlsanity.com/paper/2307.04231/) (2023)
- [Self-Supervised Learning with Multi-View Rendering for 3D Point Cloud Analysis](https://mlsanity.com/paper/2210.15904/) (2022)
- [PointCMC: Cross-Modal Multi-Scale Correspondences Learning for Point Cloud Understanding](https://mlsanity.com/paper/2211.12032/) (2022)
- [Self-Supervised Feature Learning from Partial Point Clouds via Pose Disentanglement](https://mlsanity.com/paper/2201.03018/) (2022)
- [Masked Clustering Prediction for Unsupervised Point Cloud Pre-training](https://mlsanity.com/paper/2508.08910/) (2025)
- [ViFiCon: Vision and Wireless Association Via Self-Supervised Contrastive Learning](https://mlsanity.com/paper/2210.05513/) (2022)
- [2D-3D Attention and Entropy for Pose Robust 2D Facial Recognition](https://mlsanity.com/paper/2505.09073/) (2025)
- [A Cross Branch Fusion-Based Contrastive Learning Framework for Point Cloud Self-supervised Learning](https://mlsanity.com/paper/2505.24641/) (2025)
- [Pluggable Weakly-Supervised Cross-View Learning for Accurate Vehicle Re-Identification](https://mlsanity.com/paper/2103.05376/) (2021)
- [How do Cross-View and Cross-Modal Alignment Affect Representations in Contrastive Learning?](https://mlsanity.com/paper/2211.13309/) (2022)
- [Self-Supervised Learning for Point Clouds Data: A Survey](https://mlsanity.com/paper/2305.11881/) (2023)

## Abstract (arXiv)

The success of supervised learning requires large-scale ground truth labels which are very expensive, time-consuming, or may need special skills to annotate. To address this issue, many self- or un-supervised methods are developed. Unlike most existing self-supervised methods to learn only 2D image features or only 3D point cloud features, this paper presents a novel and effective self-supervised learning approach to jointly learn both 2D image features and 3D point cloud features by exploiting cross-modality and cross-view correspondences without using any human annotated labels. Specifically, 2D image features of rendered images from different views are extracted by a 2D convolutional neural network, and 3D point cloud features are extracted by a graph convolution neural network. Two types of features are fed into a two-layer fully connected neural network to estimate the cross-modality correspondence. The three networks are jointly trained (i.e. cross-modality) by verifying whether two sampled data of different modalities belong to the same object, meanwhile, the 2D convolutional neural network is additionally optimized through minimizing intra-object distance while maximizing inter-object distance of rendered images in different views (i.e. cross-view). The effectiveness of the learned 2D and 3D features is evaluated by transferring them on five different tasks including multi-view 2D shape recognition, 3D shape recognition, multi-view 2D shape retrieval, 3D shape retrieval, and 3D part-segmentation. Extensive evaluations on all the five different tasks across different datasets demonstrate strong generalization and effectiveness of the learned 2D and 3D features by the proposed self-supervised method.