mlsanityData to 14 Oct 2026

EnSiam: Self-Supervised Learning With Ensemble Representations

arXiv 2305.13391 ↗First posted 22 May 2023cs.CV cs.LGcited by 0

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Recently, contrastive self-supervised learning, where the proximity of representations is determined based on the identities of samples, has made remarkable progress in unsupervised representation learning. SimSiam is a well-known example in this area, known for its simplicity yet powerful performance. However, it is known to be sensitive to changes in training configurations, such as hyperparameters and augmentation settings, due to its structural characteristics. To address this issue, we focus on the similarity between contrastive learning and the teacher-student framework in knowledge distillation. Inspired by the ensemble-based knowledge distillation approach, the proposed method, EnSiam, aims to improve the contrastive learning procedure using ensemble representations. This can provide stable pseudo labels, providing better performance. Experiments demonstrate that EnSiam outperforms previous state-of-the-art methods in most cases, including the experiments on ImageNet, which shows that EnSiam is capable of learning high-quality representations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 45 of its 60 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

45 of 60 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  9. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  10. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  11. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  12. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
33 more
  1. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  2. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  3. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  4. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  5. 2019Contrastive Multiview Codingcited by 1,301
  6. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  7. 2016Colorful Image Colorizationcited by 1,114
  8. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  9. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  10. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  11. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  12. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  13. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 640
  14. 2019Self-labelling via simultaneous clustering and representation learningcited by 528
  15. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  16. 2020Hard Negative Mixing for Contrastive Learningcited by 402
  17. 2020Whitening for Self-Supervised Representation Learningcited by 279
  18. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 238
  19. 2016Joint Unsupervised Learning of Deep Representations and Image Clusterscited by 205
  20. 2019Local Aggregation for Unsupervised Learning of Visual Embeddingscited by 204
  21. 2017Colorization as a Proxy Task for Visual Understandingcited by 175
  22. 2020Self-Knowledge Distillation with Progressive Refinement of Targetscited by 107
  23. 2021Contrastive Learning with Stronger Augmentationscited by 100
  24. 2020MS2L: Multi-Task Self-Supervised Learning for Skeleton Based Action Recognitioncited by 92
  25. 2020AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversariescited by 90
  26. 2021ReSSL: Relational Self-Supervised Learning with Weak Augmentationcited by 84
  27. 2021Weakly Supervised Contrastive Learningcited by 81
  28. 2018Learning Image Representations by Completing Damaged Jigsaw Puzzlescited by 66
  29. 2018Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Studentscited by 50
  30. 2019Ensemble Knowledge Distillation for Learning Improved and Efficient Networkscited by 46
  31. 2021Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwritingcited by 38
  32. 2021Directional Self-supervised Learning for Heavy Image Augmentationscited by 11
  33. 2022TransRank: Self-supervised Video Representation Learning via Ranking-based Transformation Recognitioncited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Discriminative and Consistent Representation Distillation
  2. 2021Hierarchical Self-supervised Augmented Knowledge Distillation
  3. 2024Preview-based Category Contrastive Learning for Knowledge Distillation
  4. 2025Enriching Knowledge Distillation with Intra-Class Contrastive Learning
  5. 2021Boosting Contrastive Learning with Relation Knowledge Distillation
  6. 2023MV-MR: multi-views and multi-representations for self-supervised learning and knowledge distillation
  7. 2023DistillCSE: Distilled Contrastive Learning for Sentence Embeddings
  8. 2023MOMA:Distill from Self-Supervised Teachers