mlsanityData to 14 Oct 2026

Learning deep representations by mutual information estimation and maximization

arXiv 1808.06670 ↗First posted 20 Aug 2018stat.ML cs.LGcited by 1,273

Not on the map. Why?

Cited by 23 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this work, we perform unsupervised learning of representations by maximizing mutual information between an input and the output of a deep neural network encoder. Importantly, we show that structure matters: incorporating knowledge about locality of the input to the objective can greatly influence a representation's suitability for downstream tasks. We further control characteristics of the representation by matching to a prior distribution adversarially. Our method, which we call Deep InfoMax (DIM), outperforms a number of popular unsupervised learning methods and competes with fully-supervised learning on several classification tasks. DIM opens new avenues for unsupervised learning of representations and is an important step towards flexible formulations of representation-learning objectives for specific end-goals.

Key facts. Cited by 1,273 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Aug 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 359; 2022: 306; 2023: 223; 2024: 178; 2025: 128; 2026: 78.

1,273 papers.

Sort:

By area:

  1. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  2. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  3. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  4. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  5. 2021Time-Series Representation Learning via Temporal and Contextual Contrastingcited by 345
  6. 2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
  7. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  8. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 312
  9. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 306
  10. 2021Graph Self-Supervised Learning: A Surveycited by 289
  11. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 254
  12. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
38 more
  1. 2023A Cookbook of Self-Supervised Learningcited by 250
  2. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  3. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  4. 2021Exploring Cross-Image Pixel Contrast for Semantic Segmentationcited by 242
  5. 2022Unsupervised Semantic Segmentation by Distilling Feature Correspondencescited by 241
  6. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  7. 2021Contrastive Learning Inverts the Data Generating Processcited by 221
  8. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  9. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  10. 2021Adversarial Graph Augmentation to Improve Graph Contrastive Learningcited by 200
  11. 2021Large-Scale Representation Learning on Graphs via Bootstrappingcited by 186
  12. 2021Self-Supervised Learning of Graph Neural Networks: A Unified Reviewcited by 184
  13. 2021Spatio-temporal Self-Supervised Representation Learning for 3D Point Cloudscited by 180
  14. 2021Anomaly Detection on Attributed Networks via Contrastive Self-Supervised Learningcited by 177
  15. 2021FSCE: Few-Shot Object Detection via Contrastive Proposal Encodingcited by 168
  16. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 164
  17. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 163
  18. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  19. 2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 156
  20. 2021Model Adaptation: Historical Contrastive Learning for Unsupervised Domain Adaptation without Source Datacited by 152
  21. 2021Efficient Visual Pretraining with Contrastive Detectioncited by 150
  22. 2021Efficient Self-supervised Vision Transformers for Representation Learningcited by 149
  23. 20213D Infomax improves GNNs for Molecular Property Predictioncited by 146
  24. 2021Decoupled Contrastive Learningcited by 143
  25. 2022Federated Learning from Pre-Trained Models: A Contrastive Learning Approachcited by 142
  26. 2021From Canonical Correlation Analysis to Self-supervised Graph Neural Networkscited by 141
  27. 2022Rethinking Semantic Segmentation: A Prototype Viewcited by 140
  28. 2022Self-supervised Learning in Remote Sensing: A Reviewcited by 137
  29. 2022Contrastive Masked Autoencoders are Stronger Vision Learnerscited by 133
  30. 2021Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysiscited by 131
  31. 2022A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Languagecited by 131
  32. 2021Multimodal Contrastive Training for Visual Representation Learningcited by 124
  33. 2021BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learningcited by 123
  34. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 122
  35. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  36. 2021Causality-inspired Single-source Domain Generalization for Medical Image Segmentationcited by 119
  37. 2022VICRegL: Self-Supervised Learning of Local Visual Featurescited by 118
  38. 2021MST: Masked Self-Supervised Transformer for Visual Representationcited by 114

50 most cited of 1,273 All 1,273 (JSON)