# P4Contrast: Contrastive Learning with Pairs of Point-Pixel Pairs for RGB-D Scene Understanding

arXiv: https://arxiv.org/abs/2012.13089 · HTML: https://mlsanity.com/paper/2012.13089/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Dec 2020.

## Built on by (most cited first, 30 of 41)

- [CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completion](https://mlsanity.com/paper/2210.10716/) (2022)
- [Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Data](https://mlsanity.com/paper/2203.16258/) (2022)
- [Multi-modal Sensor Fusion for Auto Driving Perception: A Survey](https://mlsanity.com/paper/2202.02703/) (2022)
- [SQN: Weakly-Supervised Semantic Segmentation of Large-Scale 3D Point Clouds](https://mlsanity.com/paper/2104.04891/) (2021)
- [Contrastive Boundary Learning for Point Cloud Segmentation](https://mlsanity.com/paper/2203.05272/) (2022)
- [PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detection](https://mlsanity.com/paper/2303.08129/) (2023)
- [Contrastive Multimodal Fusion with TupleInfoNCE](https://mlsanity.com/paper/2107.02575/) (2021)
- [Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learning](https://mlsanity.com/paper/2206.10442/) (2022)
- [GrowSP: Unsupervised Semantic Segmentation of 3D Point Clouds](https://mlsanity.com/paper/2305.16404/) (2023)
- [CAT-Det: Contrastively Augmented Transformer for Multi-modal 3D Object Detection](https://mlsanity.com/paper/2204.00325/) (2022)
- [Complete-to-Partial 4D Distillation for Self-Supervised Point Cloud Sequence Representation Learning](https://mlsanity.com/paper/2212.05330/) (2022)
- [Versatile Multi-Modal Pre-Training for Human-Centric Perception](https://mlsanity.com/paper/2203.13815/) (2022)
- [A Closer Look at Invariances in Self-supervised Pre-training for 3D Vision](https://mlsanity.com/paper/2207.04997/) (2022)
- [VIBUS: Data-efficient 3D Scene Parsing with VIewpoint Bottleneck and Uncertainty-Spectrum Modeling](https://mlsanity.com/paper/2210.11472/) (2022)
- [DetMatch: Two Teachers are Better Than One for Joint 2D and 3D Semi-Supervised Object Detection](https://mlsanity.com/paper/2203.09510/) (2022)
- [LiDAL: Inter-frame Uncertainty Based Active Learning for 3D LiDAR Semantic Segmentation](https://mlsanity.com/paper/2211.05997/) (2022)
- [CO^3: Cooperative Unsupervised 3D Representation Learning for Autonomous Driving](https://mlsanity.com/paper/2206.04028/) (2022)
- [COARSE3D: Class-Prototypes for Contrastive Learning in Weakly-Supervised 3D Point Cloud Segmentation](https://mlsanity.com/paper/2210.01784/) (2022)
- [PCR-CG: Point Cloud Registration via Deep Explicit Color and Geometry](https://mlsanity.com/paper/2302.14418/) (2023)
- [Multilingual Molecular Representation Learning via Contrastive Pre-training](https://mlsanity.com/paper/2109.08830/) (2021)
- [Pointly-supervised 3D Scene Parsing with Viewpoint Bottleneck](https://mlsanity.com/paper/2109.08553/) (2021)
- [PointMatch: A Consistency Training Framework for Weakly Supervised Semantic Segmentation of 3D Point Clouds](https://mlsanity.com/paper/2202.10705/) (2022)
- [Self-Supervised Image Representation Learning with Geometric Set Consistency](https://mlsanity.com/paper/2203.15361/) (2022)
- [Robots Autonomously Detecting People: A Multimodal Deep Contrastive Learning Method Robust to Intraclass Variations](https://mlsanity.com/paper/2203.00187/) (2022)
- [Learning Dense Object Descriptors from Multiple Views for Low-shot Category Generalization](https://mlsanity.com/paper/2211.15059/) (2022)
- [CrossVideo: Self-supervised Cross-modal Contrastive Learning for Point Cloud Video Understanding](https://mlsanity.com/paper/2401.09057/) (2024)
- [PointCMC: Cross-Modal Multi-Scale Correspondences Learning for Point Cloud Understanding](https://mlsanity.com/paper/2211.12032/) (2022)
- [Pre-Training LiDAR-Based 3D Object Detectors Through Colorization](https://mlsanity.com/paper/2310.14592/) (2023)
- [Information-guided pixel augmentation for pixel-wise contrastive learning](https://mlsanity.com/paper/2211.07118/) (2022)
- [On Geometry-Enhanced Parameter-Efficient Fine-Tuning for 3D Scene Segmentation](https://mlsanity.com/paper/2505.22444/) (2025)

## Abstract (arXiv)

Self-supervised representation learning is a critical problem in computer vision, as it provides a way to pretrain feature extractors on large unlabeled datasets that can be used as an initialization for more efficient and effective training on downstream tasks. A promising approach is to use contrastive learning to learn a latent space where features are close for similar data samples and far apart for dissimilar ones. This approach has demonstrated tremendous success for pretraining both image and point cloud feature extractors, but it has been barely investigated for multi-modal RGB-D scans, especially with the goal of facilitating high-level scene understanding. To solve this problem, we propose contrasting "pairs of point-pixel pairs", where positives include pairs of RGB-D points in correspondence, and negatives include pairs where one of the two modalities has been disturbed and/or the two RGB-D points are not in correspondence. This provides extra flexibility in making hard negatives and helps networks to learn features from both modalities, not just the more discriminating one of the two. Experiments show that this proposed approach yields better performance on three large-scale RGB-D scene understanding benchmarks (ScanNet, SUN RGB-D, and 3RScan) than previous pretraining approaches.