# Self-supervised Representation Learning with Relative Predictive Coding

arXiv: https://arxiv.org/abs/2103.11275 · HTML: https://mlsanity.com/paper/2103.11275/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 65 references that are in the map.
- First posted on arXiv 21 Mar 2021.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Deep Graph Infomax](https://mlsanity.com/paper/1809.10341/) (2018)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [Selective Kernel Networks](https://mlsanity.com/paper/1903.06586/) (2019)
- [Debiased Contrastive Learning](https://mlsanity.com/paper/2007.00224/) (2020)
- [On Mutual Information Maximization for Representation Learning](https://mlsanity.com/paper/1907.13625/) (2019)
- [Contrastive Learning of Structured World Models](https://mlsanity.com/paper/1911.12247/) (2019)

## Built on by (most cited first, 28 of 28)

- [Decoupled Contrastive Learning](https://mlsanity.com/paper/2110.06848/) (2021)
- [Lossy Compression for Lossless Prediction](https://mlsanity.com/paper/2106.10800/) (2021)
- [On the Stepwise Nature of Self-Supervised Learning](https://mlsanity.com/paper/2303.15438/) (2023)
- [Revitalizing CNN Attentions via Transformers in Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/2110.05340/) (2021)
- [Self-Supervised Learning via Maximum Entropy Coding](https://mlsanity.com/paper/2210.11464/) (2022)
- [A Note on Connecting Barlow Twins with Negative-Sample-Free Contrastive Learning](https://mlsanity.com/paper/2104.13712/) (2021)
- [Conditional Contrastive Learning with Kernel](https://mlsanity.com/paper/2202.05458/) (2022)
- [Understanding Contrastive Learning via Distributionally Robust Optimization](https://mlsanity.com/paper/2310.11048/) (2023)
- [What shapes the loss landscape of self-supervised learning?](https://mlsanity.com/paper/2210.00638/) (2022)
- [Soft Neighbors are Positive Supporters in Contrastive Visual Representation Learning](https://mlsanity.com/paper/2303.17142/) (2023)
- [Efficient Non-Parametric Uncertainty Quantification for Black-Box Large Language Models and Decision Planning](https://mlsanity.com/paper/2402.00251/) (2024)
- [f-MICL: Understanding and Generalizing InfoNCE-based Contrastive Learning](https://mlsanity.com/paper/2402.10150/) (2024)
- [TiCo: Transformation Invariance and Covariance Contrast for Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/2206.10698/) (2022)
- [Sample and Predict Your Latent: Modality-free Sequential Disentanglement via Contrastive Estimation](https://mlsanity.com/paper/2305.15924/) (2023)
- [RenyiCL: Contrastive Representation Learning with Skew Renyi Divergence](https://mlsanity.com/paper/2208.06270/) (2022)
- [Towards an Improved Understanding and Utilization of Maximum Manifold Capacity Representations](https://mlsanity.com/paper/2406.09366/) (2024)
- [Integrating Auxiliary Information in Self-supervised Learning](https://mlsanity.com/paper/2106.02869/) (2021)
- [Learning Weakly-Supervised Contrastive Representations](https://mlsanity.com/paper/2202.06670/) (2022)
- [Towards Understanding the Mechanism of Contrastive Learning via Similarity Structure: A Theoretical Analysis](https://mlsanity.com/paper/2304.00395/) (2023)
- [Unsupervised Reinforcement Learning for Transferable Manipulation Skill Discovery](https://mlsanity.com/paper/2204.13906/) (2022)
- [Self-Supervised Representation Learning for Adversarial Attack Detection](https://mlsanity.com/paper/2407.04382/) (2024)
- [MUC: Machine Unlearning for Contrastive Learning with Black-box Evaluation](https://mlsanity.com/paper/2406.03603/) (2024)
- [Exploiting Supervised Poison Vulnerability to Strengthen Self-Supervised Defense](https://mlsanity.com/paper/2409.08509/) (2024)
- [An Attention-based Framework for Fair Contrastive Learning](https://mlsanity.com/paper/2411.14765/) (2024)
- [A Deep Bayesian Nonparametric Framework for Robust Mutual Information Estimation](https://mlsanity.com/paper/2503.08902/) (2025)
- [Understanding InfoNCE: Transition Probability Matrix Induced Feature Clustering](https://mlsanity.com/paper/2511.12180/) (2025)
- [Self-Predictive Representation for Autonomous UAV Object-Goal Navigation](https://mlsanity.com/paper/2604.21130/) (2026)
- [HOLA: Holistic Multi-Modal Alignment for Open-Set 3D Recognition](https://mlsanity.com/paper/2606.01334/) (2026)

## Abstract (arXiv)

This paper introduces Relative Predictive Coding (RPC), a new contrastive representation learning objective that maintains a good balance among training stability, minibatch size sensitivity, and downstream task performance. The key to the success of RPC is two-fold. First, RPC introduces the relative parameters to regularize the objective for boundedness and low variance. Second, RPC contains no logarithm and exponential score functions, which are the main cause of training instability in prior contrastive objectives. We empirically verify the effectiveness of RPC on benchmark vision and speech self-supervised learning tasks. Lastly, we relate RPC with mutual information (MI) estimation, showing RPC can be used to estimate MI with low variance.