mlsanityData to 7 Oct 2026

MC-SSL0.0: Towards Multi-Concept Self-Supervised Learning

arXiv 2111.15340 ↗First posted 30 Nov 2021cs.CV cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Self-supervised pretraining is the method of choice for natural language processing models and is rapidly gaining popularity in many vision tasks. Recently, self-supervised pretraining has shown to outperform supervised pretraining for many downstream vision applications, marking a milestone in the area. This superiority is attributed to the negative impact of incomplete labelling of the training images, which convey multiple concepts, but are annotated using a single dominant class label. Although Self-Supervised Learning (SSL), in principle, is free of this limitation, the choice of pretext task facilitating SSL is perpetuating this shortcoming by driving the learning process towards a single concept output. This study aims to investigate the possibility of modelling all the concepts present in an image without using labels. In this aspect the proposed SSL frame-work MC-SSL0.0 is a step towards Multi-Concept Self-Supervised Learning (MC-SSL) that goes beyond modelling single dominant label in an image to effectively utilise the information from all the concepts present in it. MC-SSL0.0 consists of two core design concepts, group masked model learning and learning of pseudo-concept for data token using a momentum encoder (teacher-student) framework. The experimental results on multi-label and multi-class image classification downstream tasks demonstrate that MC-SSL0.0 not only surpasses existing SSL methods but also outperforms supervised transfer learning. The source code will be made publicly available for community to train on bigger corpus.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 49 of its 60 references that are in the map. First posted on arXiv 30 Nov 2021.

Builds on

49 of 60 references.

References · 49

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
37 more
  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  2. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  3. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  6. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  7. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  8. 2017Decoupled Weight Decay Regularizationcited by 1,722
  9. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  10. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  11. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  12. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  13. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  14. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  15. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  16. 2016Colorful Image Colorizationcited by 1,113
  17. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  18. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  19. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  20. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  21. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  22. 2020Asymmetric Loss For Multi-Label Classificationcited by 379
  23. 2016Learning Representations for Automatic Colorizationcited by 251
  24. 2019Large Scale Adversarial Representation Learningcited by 238
  25. 2016Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Predictioncited by 217
  26. 2016Adversarially Learned Inferencecited by 216
  27. 2017Colorization as a Proxy Task for Visual Understandingcited by 175
  28. 2020General Multi-label Image Classification with Transformerscited by 126
  29. 2021Query2Label: A Simple Transformer Way to Multi-Label Classificationcited by 126
  30. 2019Learning Semantic-Specific Graph Representation for Multi-Label Image Recognitioncited by 116
  31. 2021SiT: Self-supervised vIsion Transformercited by 99
  32. 2017Unsupervised Learning by Predicting Noisecited by 97
  33. 2020Knowledge-Guided Multi-Label Few-Shot Learning for General Image Recognitioncited by 69
  34. 2018Learning Image Representations by Completing Damaged Jigsaw Puzzlescited by 66
  35. 2018Self-Supervised Feature Learning by Learning to Spot Artifactscited by 43
  36. 2020Self-Supervised Relational Reasoning for Representation Learningcited by 37
  37. 2021MlTr: Multi-label Classification with Transformercited by 21

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 1; 2024: 3; 2025: 1.

10 papers.

  1. 2022SemMAE: Semantic-Guided Masking for Learning Masked Autoencoderscited by 113
  2. 2022ASiT: Local-Global Audio Spectrogram vIsion Transformer for Event Classificationcited by 13
  3. 2022GMML is All you Needcited by 12
  4. 2022SB-SSL: Slice-Based Self-Supervised Transformers for Knee Abnormality Classification from MRIcited by 6
  5. 2022SPCXR: Self-supervised Pretraining using Chest X-rays Towards a Domain Specific Foundation Modelcited by 6
  6. 2023Masked Momentum Contrastive Learning for Zero-shot Semantic Understandingcited by 4
  7. 2024DiCoM -- Diverse Concept Modeling towards Enhancing Generalizability in Chest X-Ray Studiescited by 4
  8. 2025Self-Guided Masked Autoencodercited by 3
  9. 2024Investigating Self-Supervised Methods for Label-Efficient Learningcited by 2
  10. 2024Pseudo Labelling for Enhanced Masked Autoencoderscited by 1