mlsanityData to 7 Oct 2026

Compressive Visual Representations

arXiv 2109.12909 ↗First posted 27 Sep 2021cs.LG cs.CV cs.ITcited by 41

Not on the map. Why?

Abstract (arXiv)

Learning effective visual representations that generalize well without human supervision is a fundamental problem in order to apply Machine Learning to a wide variety of tasks. Recently, two families of self-supervised methods, contrastive learning and latent bootstrapping, exemplified by SimCLR and BYOL respectively, have made significant progress. In this work, we hypothesize that adding explicit information compression to these algorithms yields better and more robust representations. We verify this by developing SimCLR and BYOL formulations compatible with the Conditional Entropy Bottleneck (CEB) objective, allowing us to both measure and control the amount of compression in the learned representation, and observe their impact on downstream tasks. Furthermore, we explore the relationship between Lipschitz continuity and compression, showing a tractable lower bound on the Lipschitz constant of the encoders we learn. As Lipschitz continuity is closely related to robustness, this provides a new explanation for why compressed models are more robust. Our experiments confirm that adding compression to SimCLR and BYOL significantly improves linear evaluation accuracies and model robustness across a wide range of domain shifts. In particular, the compressed version of BYOL achieves 76.0% Top-1 linear evaluation accuracy on ImageNet with ResNet-50, and 78.8% with ResNet-50 2x.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 55 of its 80 references that are in the map. First posted on arXiv 27 Sep 2021.

Builds on

55 of 80 references.

References · 55

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  10. 2013Auto-Encoding Variational Bayescited by 3,980
  11. 2016Wide Residual Networkscited by 3,057
  12. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
38 more
  1. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  2. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  3. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  4. 2013Describing Textures in the Wildcited by 2,110
  5. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  6. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  7. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  8. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  9. 2019Natural Adversarial Examplescited by 1,401
  10. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  11. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  12. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  13. 2019Contrastive Multiview Codingcited by 1,297
  14. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  15. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  16. 2016Deep Variational Information Bottleneckcited by 915
  17. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  18. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  19. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  20. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  21. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  22. 2018Do Better ImageNet Models Transfer Better?cited by 560
  23. 2019On Variational Bounds of Mutual Informationcited by 550
  24. 2017Large Batch Training of Convolutional Networkscited by 529
  25. 2019S4L: Self-Supervised Semi-Supervised Learningcited by 335
  26. 2019Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networkscited by 290
  27. 2019Revisiting Self-Supervised Visual Representation Learningcited by 250
  28. 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 213
  29. 2020Learning Robust Representations via Multi-View Information Bottleneckcited by 189
  30. 2018Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approachcited by 181
  31. 2016Information Dropout: Learning Optimal Representations Through Noisy Computationcited by 131
  32. 2017YouTube-BoundingBoxes: A Large High-Precision Human-Annotated Data Set for Object Detection in Videocited by 109
  33. 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 101
  34. 2020On Robustness and Transferability of Convolutional Neural Networkscited by 98
  35. 2020The Conditional Entropy Bottleneckcited by 75
  36. 2019Do Image Classifiers Generalize Across Time?cited by 59
  37. 2020Predictive Information Accelerates Learning in RLcited by 56
  38. 2021Lossy Compression for Lossless Predictioncited by 51

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 16; 2023: 13; 2024: 7; 2025: 3; 2026: 2.

41 papers.

Sort:

By area:

  1. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 145
  2. 2022VICRegL: Self-Supervised Learning of Local Visual Featurescited by 118
  3. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 107
  4. 2023How Does Information Bottleneck Help Deep Learning?cited by 92
  5. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  6. 2022Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?cited by 67
  7. 2023LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matchingcited by 60
  8. 2023MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Featurescited by 53
  9. 2022Audiovisual Masked Autoencoderscited by 47
  10. 2023Self-supervised learning of Split Invariant Equivariant representationscited by 41
  11. 2023Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Trainingcited by 39
  12. 2022Self-Supervised Learning via Maximum Entropy Codingcited by 33
29 more
  1. 2022Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Supervised Learningcited by 25
  2. 2023An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularizationcited by 25
  3. 2022Simplicial Embeddings in Self-Supervised Learning and Downstream Classificationcited by 23
  4. 2022Weighted Ensemble Self-Supervised Learningcited by 19
  5. 2024An Information Bottleneck Perspective for Effective Noise Filtering on Retrieval-Augmented Generationcited by 16
  6. 2022What Do We Maximize in Self-Supervised Learning?cited by 15
  7. 2024Demonstrating and Reducing Shortcuts in Vision-Language Representation Learningcited by 14
  8. 2024DINO as a von Mises-Fisher mixture modelcited by 14
  9. 2024An Information Criterion for Controlled Disentanglement of Multimodal Datacited by 14
  10. 2025Mitigating Visual Knowledge Forgetting in MLLM Instruction-tuning via Modality-decoupled Gradient Descentcited by 13
  11. 2022RenyiCL: Contrastive Representation Learning with Skew Renyi Divergencecited by 8
  12. 2023Learning Anatomically Consistent Embedding for Chest Radiographycited by 8
  13. 2023SemPPL: Predicting pseudo-labels for better contrastive representationscited by 6
  14. 2025BottleHumor: Self-Informed Humor Explanation using the Information Bottleneck Principlecited by 6
  15. 2022PI-QT-Opt: Predictive Information Improves Multi-Task Robotic Reinforcement Learning at Scalecited by 5
  16. 2022Masked prediction tasks: a parameter identifiability viewcited by 4
  17. 2022No Shifted Augmentations (NSA): compact distributions for robust self-supervised Anomaly Detectioncited by 3
  18. 2022Self-Supervised Pyramid Representation Learning for Multi-Label Visual Analysis and Beyondcited by 3
  19. 2023Towards Understanding the Effect of Pretraining Label Granularitycited by 3
  20. 2023Understanding the limitations of self-supervised learning for tabular anomaly detectioncited by 3
  21. 2024How Does Distribution Matching Help Domain Generalization: An Information-theoretic Analysiscited by 3
  22. 2024Steinmetz Neural Networks for Complex-Valued Datacited by 2
  23. 2024Resource Efficient Multi-stain Kidney Glomeruli Segmentation via Self-supervisioncited by 2
  24. 2023DrasCLR: A Self-supervised Framework of Learning Disease-related and Anatomy-specific Representation for 3D Medical Imagescited by 1
  25. 2025ACE: Anatomically Consistent Embeddings in Composition and Decompositioncited by 1
  26. 2022Using Representation Expressiveness and Learnability to Evaluate Self-Supervised Learning Methodscited by 0
  27. 2023Unveiling the Potential of Probabilistic Embeddings in Self-Supervised Learningcited by 0
  28. 2026Self-Supervised Learning with a Multi-Task Latent Space Objectivecited by 0
  29. 2026Rethinking the Information Bottleneck: Structured Decomposition under Label-Induced Partitionscited by 0