mlsanityData to 14 Oct 2026

GEDI: GEnerative and DIscriminative Training for Self-Supervised Learning

arXiv 2212.13425 ↗First posted 27 Dec 2022cs.LG cs.AI cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning is a popular and powerful method for utilizing large amounts of unlabeled data, for which a wide variety of training objectives have been proposed in the literature. In this study, we perform a Bayesian analysis of state-of-the-art self-supervised learning objectives and propose a unified formulation based on likelihood learning. Our analysis suggests a simple method for integrating self-supervised learning with generative models, allowing for the joint training of these two seemingly distinct approaches. We refer to this combined framework as GEDI, which stands for GEnerative and DIscriminative training. Additionally, we demonstrate an instantiation of the GEDI framework by integrating an energy-based model with a cluster-based self-supervised learning model. Through experiments on synthetic and real-world data, including SVHN, CIFAR10, and CIFAR100, we show that GEDI outperforms existing self-supervised learning strategies in terms of clustering performance by a wide margin. We also demonstrate that GEDI can be integrated into a neural-symbolic framework to address tasks in the small data regime, where it can use logical constraints to further improve clustering and classification performance.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 57 of its 77 references that are in the map. First posted on arXiv 27 Dec 2022.

Builds on

57 of 77 references.

References · 57

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  5. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  6. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  7. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  8. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  9. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  10. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  11. 2020Self-supervised Learning: Generative or Contrastivecited by 694
  12. 2018MINE: Mutual Information Neural Estimationcited by 683
38 more
  1. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  2. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 657
  3. 2019On Variational Bounds of Mutual Informationcited by 559
  4. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  5. 2019Self-labelling via simultaneous clustering and representation learningcited by 528
  6. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 514
  7. 2018Do Deep Generative Models Know What They Don't Know?cited by 443
  8. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 405
  9. 2019Neural Probabilistic Logic Programming in DeepProbLogcited by 394
  10. 2019On Mutual Information Maximization for Representation Learningcited by 316
  11. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  12. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 312
  13. 2020Whitening for Self-Supervised Representation Learningcited by 279
  14. 2017A Semantic Loss Function for Deep Learning with Symbolic Knowledgecited by 270
  15. 2020Contrastive Clusteringcited by 265
  16. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  17. 2020Representation Learning via Invariant Causal Mechanismscited by 199
  18. 2018Formal Limitations on the Measurement of Mutual Informationcited by 189
  19. 2020Contrastive Training for Improved Out-of-Distribution Detectioncited by 186
  20. 2016A Theory of Generative ConvNetcited by 174
  21. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 149
  22. 2019Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Modelcited by 148
  23. 2019Understanding the Limitations of Variational Mutual Information Estimatorscited by 140
  24. 2022VICRegL: Self-Supervised Learning of Local Visual Featurescited by 118
  25. 2021Solo-learn: A Library of Self-supervised Methods for Visual Representation Learningcited by 116
  26. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 108
  27. 2019On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Modelscited by 105
  28. 2021Contrastive Learning with Stronger Augmentationscited by 100
  29. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  30. 2022Adversarial Masking for Self-Supervised Learningcited by 78
  31. 2022Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?cited by 67
  32. 2020No MCMC for me: Amortized sampling for fast and stable training of energy-based modelscited by 64
  33. 2021Learning Representation for Clustering via Prototype Scattering and Positive Samplingcited by 58
  34. 2020Contrastive estimation reveals topic posterior information to linear modelscited by 54
  35. 2022Neural Manifold Clustering and Embeddingcited by 44
  36. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  37. 2022Self-Supervised Learning with an Information Maximization Criterioncited by 38
  38. 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 34

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 1.

6 papers.

  1. 2023Controlled Text Generation via Language Model Arithmeticcited by 51
  2. 2023Incorporating Unlabelled Data into Bayesian Neural Networkscited by 9
  3. 2023Learning Symbolic Representations Through Joint GEnerative and DIscriminative Trainingcited by 8
  4. 2024A Probabilistic Model Behind Self-Supervised Learningcited by 5
  5. 2023The Triad of Failure Modes and a Possible Way Outcited by 3
  6. 2023Unifying Self-Supervised Clustering and Energy-Based Modelscited by 2