mlsanityData to 7 Oct 2026

Self-supervised Representation Learning with Relative Predictive Coding

arXiv 2103.11275 ↗First posted 21 Mar 2021cs.LG cs.IT math.ITcited by 27

Not on the map. Why?

Abstract (arXiv)

This paper introduces Relative Predictive Coding (RPC), a new contrastive representation learning objective that maintains a good balance among training stability, minibatch size sensitivity, and downstream task performance. The key to the success of RPC is two-fold. First, RPC introduces the relative parameters to regularize the objective for boundedness and low variance. Second, RPC contains no logarithm and exponential score functions, which are the main cause of training instability in prior contrastive objectives. We empirically verify the effectiveness of RPC on benchmark vision and speech self-supervised learning tasks. Lastly, we relate RPC with mutual information (MI) estimation, showing RPC can be used to estimate MI with low variance.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 39 of its 65 references that are in the map. First posted on arXiv 21 Mar 2021.

Builds on

39 of 65 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  10. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  11. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  12. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
27 more
  1. 2020Supervised Contrastive Learningcited by 2,803
  2. 2017Improved Training of Wasserstein GANscited by 2,223
  3. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  4. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  5. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  6. 2019Contrastive Multiview Codingcited by 1,297
  7. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  8. 2018Deep Graph Infomaxcited by 1,101
  9. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  10. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  11. 2019On Variational Bounds of Mutual Informationcited by 550
  12. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  13. 2017Large Batch Training of Convolutional Networkscited by 529
  14. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
  15. 2019Selective Kernel Networkscited by 401
  16. 2020Debiased Contrastive Learningcited by 354
  17. 2019On Mutual Information Maximization for Representation Learningcited by 315
  18. 2019Contrastive Learning of Structured World Modelscited by 206
  19. 2018Formal Limitations on the Measurement of Mutual Informationcited by 189
  20. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  21. 2019Understanding the Limitations of Variational Mutual Information Estimatorscited by 140
  22. 2020Unsupervised pretraining transfers well across languagescited by 100
  23. 2019Putting An End to End-to-End: Gradient-Isolated Learning of Representationscited by 85
  24. 2019A Mutual Information Maximization Perspective of Language Representation Learningcited by 85
  25. 2019Wasserstein Dependency Measure for Representation Learningcited by 69
  26. 2020Neural Methods for Point-wise Dependency Estimationcited by 22
  27. 2020Self-supervised Learning from a Multi-view Perspectivecited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 7; 2023: 5; 2024: 7; 2025: 2; 2026: 1.

27 papers.

Sort:

By area:

  1. 2021Decoupled Contrastive Learningcited by 141
  2. 2021Lossy Compression for Lossless Predictioncited by 51
  3. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
  4. 2021Revitalizing CNN Attentions via Transformers in Self-Supervised Visual Representation Learningcited by 34
  5. 2022Self-Supervised Learning via Maximum Entropy Codingcited by 33
  6. 2021A Note on Connecting Barlow Twins with Negative-Sample-Free Contrastive Learningcited by 26
  7. 2022Conditional Contrastive Learning with Kernelcited by 22
  8. 2023Understanding Contrastive Learning via Distributionally Robust Optimizationcited by 21
  9. 2022What shapes the loss landscape of self-supervised learning?cited by 19
  10. 2023Soft Neighbors are Positive Supporters in Contrastive Visual Representation Learningcited by 18
  11. 2024Efficient Non-Parametric Uncertainty Quantification for Black-Box Large Language Models and Decision Planningcited by 16
  12. 2024f-MICL: Understanding and Generalizing InfoNCE-based Contrastive Learningcited by 13
15 more
  1. 2022TiCo: Transformation Invariance and Covariance Contrast for Self-Supervised Visual Representation Learningcited by 12
  2. 2023Sample and Predict Your Latent: Modality-free Sequential Disentanglement via Contrastive Estimationcited by 10
  3. 2022RenyiCL: Contrastive Representation Learning with Skew Renyi Divergencecited by 8
  4. 2024Towards an Improved Understanding and Utilization of Maximum Manifold Capacity Representationscited by 8
  5. 2021Integrating Auxiliary Information in Self-supervised Learningcited by 6
  6. 2022Learning Weakly-Supervised Contrastive Representationscited by 6
  7. 2023Towards Understanding the Mechanism of Contrastive Learning via Similarity Structure: A Theoretical Analysiscited by 6
  8. 2022Unsupervised Reinforcement Learning for Transferable Manipulation Skill Discoverycited by 5
  9. 2024Self-Supervised Representation Learning for Adversarial Attack Detectioncited by 4
  10. 2024MUC: Machine Unlearning for Contrastive Learning with Black-box Evaluationcited by 1
  11. 2024Exploiting Supervised Poison Vulnerability to Strengthen Self-Supervised Defensecited by 1
  12. 2024An Attention-based Framework for Fair Contrastive Learningcited by 0
  13. 2025A Deep Bayesian Nonparametric Framework for Robust Mutual Information Estimationcited by 0
  14. 2025Understanding InfoNCE: Transition Probability Matrix Induced Feature Clusteringcited by 0
  15. 2026HOLA: Holistic Multi-Modal Alignment for Open-Set 3D Recognitioncited by 0