# Heterogeneous Contrastive Learning: Encoding Spatial Information for Compact Visual Representations

arXiv: https://arxiv.org/abs/2011.09941 · HTML: https://mlsanity.com/paper/2011.09941/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Nov 2020.

## Built on by (most cited first, 9 of 9)

- [Investigating the Role of Negatives in Contrastive Representation Learning](https://mlsanity.com/paper/2106.09943/) (2021)
- [rPPG-MAE: Self-supervised Pre-training with Masked Autoencoders for Remote Physiological Measurement](https://mlsanity.com/paper/2306.02301/) (2023)
- [Heterogeneous Contrastive Learning](https://mlsanity.com/paper/2105.09401/) (2021)
- [Semantic-Aware Generation for Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/2111.13163/) (2021)
- [Disjoint Masking with Joint Distillation for Efficient Masked Image Modeling](https://mlsanity.com/paper/2301.00230/) (2022)
- [Beyond Instance Discrimination: Relation-aware Contrastive Self-supervised Learning](https://mlsanity.com/paper/2211.01796/) (2022)
- [What Is Considered Complete for Visual Recognition?](https://mlsanity.com/paper/2105.13978/) (2021)
- [Exploring the Diversity and Invariance in Yourself for Visual Pre-Training Task](https://mlsanity.com/paper/2106.00537/) (2021)
- [Source-free Semantic Regularization Learning for Semi-supervised Domain Adaptation](https://mlsanity.com/paper/2501.01126/) (2025)

## Abstract (arXiv)

Contrastive learning has achieved great success in self-supervised visual representation learning, but existing approaches mostly ignored spatial information which is often crucial for visual representation. This paper presents heterogeneous contrastive learning (HCL), an effective approach that adds spatial information to the encoding stage to alleviate the learning inconsistency between the contrastive objective and strong data augmentation operations. We demonstrate the effectiveness of HCL by showing that (i) it achieves higher accuracy in instance discrimination and (ii) it surpasses existing pre-training methods in a series of downstream tasks while shrinking the pre-training costs by half. More importantly, we show that our approach achieves higher efficiency in visual representations, and thus delivers a key message to inspire the future research of self-supervised visual representation learning.