mlsanityData to 14 Oct 2026

Maximizing Incremental Information Entropy for Contrastive Learning

arXiv 2603.12594 ↗First posted 13 Mar 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Contrastive learning has achieved remarkable success in self-supervised representation learning, often guided by information-theoretic objectives such as mutual information maximization. Motivated by the limitations of static augmentations and rigid invariance constraints, we propose IE-CL (Incremental-Entropy Contrastive Learning), a framework that explicitly optimizes the entropy gain between augmented views while preserving semantic consistency. Our theoretical framework reframes the challenge by identifying the encoder as an information bottleneck and proposes a joint optimization of two components: a learnable transformation for entropy generation and an encoder regularizer for its preservation. Experiments on CIFAR-10/100, STL-10, and ImageNet demonstrate that IE-CL consistently improves performance under small-batch settings. Moreover, our core modules can be seamlessly integrated into existing frameworks. This work bridges theoretical principles and practice, offering a new perspective in contrastive learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 49 of its 63 references that are in the map. First posted on arXiv 13 Mar 2026.

Builds on

49 of 63 references.

References · 49

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  9. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
37 more
  1. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  2. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  3. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  4. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  5. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  6. 2019Contrastive Multiview Codingcited by 1,301
  7. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  8. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  9. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  10. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  11. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  12. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  13. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  14. 2020Keystroke Dynamics as Part of Lifeloggingcited by 334
  15. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 326
  16. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  17. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  18. 2020Whitening for Self-Supervised Representation Learningcited by 279
  19. 2021Contrastive Learning Inverts the Data Generating Processcited by 221
  20. 2021Anomaly Detection on Attributed Networks via Contrastive Self-Supervised Learningcited by 177
  21. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  22. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 108
  23. 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
  24. 2022Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptationcited by 67
  25. 2022Self-Supervised Learning with an Information Maximization Criterioncited by 38
  26. 2022Understanding Deep Contrastive Learning via Coordinate-wise Optimizationcited by 35
  27. 2022Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embeddingcited by 28
  28. 2023Contrastive Learning Is Spectral Clustering On Similarity Graphcited by 25
  29. 2022Extreme Masking for Learning Instance and Distributed Visual Representationscited by 21
  30. 2023Matrix Information Theory for Self-Supervised Learningcited by 21
  31. 2023EMP-SSL: Towards Self-Supervised Learning in One Training Epochcited by 19
  32. 2022MimCo: Masked Image Modeling Pre-training with Contrastive Teachercited by 16
  33. 2023All4One: Symbiotic Neighbour Contrastive Learning via Self-Attention and Redundancy Reductioncited by 11
  34. 2025MedKAN: An Advanced Kolmogorov-Arnold Network for Medical Image Classificationcited by 9
  35. 2022Revisiting the Critical Factors of Augmentation-Invariant Representation Learningcited by 6
  36. 2022Establishing a stronger baseline for lightweight contrastive modelscited by 3
  37. 2026Vision KAN: Towards an Attention-Free Backbone for Vision with Kolmogorov-Arnold Networkscited by 2

Built on by

Not cited yet.