mlsanityData to 14 Oct 2026

Cross-Architectural Positive Pairs improve the effectiveness of Self-Supervised Learning

arXiv 2301.12025 ↗First posted 27 Jan 2023cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Existing self-supervised techniques have extreme computational requirements and suffer a substantial drop in performance with a reduction in batch size or pretraining epochs. This paper presents Cross Architectural - Self Supervision (CASS), a novel self-supervised learning approach that leverages Transformer and CNN simultaneously. Compared to the existing state-of-the-art self-supervised learning approaches, we empirically show that CASS-trained CNNs and Transformers across four diverse datasets gained an average of 3.8% with 1% labeled data, 5.9% with 10% labeled data, and 10.13% with 100% labeled data while taking 69% less time. We also show that CASS is much more robust to changes in batch size and training epochs than existing state-of-the-art self-supervised learning approaches. We have open-sourced our code at https://github.com/pranavsinghps1/CASS.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 53 references that are in the map. First posted on arXiv 27 Jan 2023.

Builds on

34 of 53 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  11. 2022A ConvNet for the 2020scited by 3,579
  12. 2020Exploring Simple Siamese Representation Learningcited by 2,866
22 more
  1. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  2. 2020Knowledge Distillation: A Surveycited by 1,603
  3. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  4. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  5. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  6. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  7. 2017Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)cited by 549
  8. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  9. 2019Transfusion: Understanding Transfer Learning for Medical Imagingcited by 368
  10. 2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
  11. 2022Masked Siamese Networks for Label-Efficient Learningcited by 302
  12. 2019BCN20000: Dermoscopic Lesions in the Wildcited by 225
  13. 2019A Survey of the Recent Architectures of Deep Convolutional Neural Networkscited by 213
  14. 2021Torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer visioncited by 81
  15. 2021Is it Time to Replace CNNs with Transformers for Medical Images?cited by 60
  16. 2022The Hidden Uniform Cluster Prior in Self-Supervised Learningcited by 58
  17. 2019Skin Lesion Classification Using Ensembles of Multi-Resolution EfficientNets with Meta Datacited by 43
  18. 2021COVID-19 Prognosis via Self-Supervised Representation Learning and Multi-Image Predictioncited by 36
  19. 2022Self-supervised Learning from 100 Million Medical Imagescited by 31
  20. 2022Cross-Architecture Self-supervised Video Representation Learningcited by 13
  21. 2022CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classificationcited by 12
  22. 2022A Data-Efficient Deep Learning Framework for Segmentation and Classification of Histopathology Imagescited by 6

Built on by

Not cited yet.