mlsanityData to 7 Oct 2026

CompRess: Self-Supervised Learning by Compressing Representations

arXiv 2010.14713 ↗First posted 28 Oct 2020cs.CV cs.LGcited by 61

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning aims to learn good representations with unlabeled data. Recent works have shown that larger models benefit more from self-supervised learning than smaller models. As a result, the gap between supervised and self-supervised learning has been greatly reduced for larger models. In this work, instead of designing a new pseudo task for self-supervised learning, we develop a model compression method to compress an already learned, deep self-supervised model (teacher) to a smaller one (student). We train the student model so that it mimics the relative similarity between the data points in the teacher's embedding space. For AlexNet, our method outperforms all previous methods including the fully supervised model on ImageNet linear evaluation (59.0% compared to 56.5%) and on nearest neighbor evaluation (50.7% compared to 41.4%). To the best of our knowledge, this is the first time a self-supervised AlexNet has outperformed supervised one on ImageNet classification. Our code is available here: https://github.com/UMBCvision/CompRess

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 19; 2023: 10; 2024: 12; 2025: 5; 2026: 1.

61 papers.

Sort:

By area:

  1. 2022SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagerycited by 402
  2. 2021SEED: Self-supervised Distillation For Visual Representationcited by 132
  3. 2023CompGS: Smaller and Faster Gaussian Splatting with Vector Quantizationcited by 127
  4. 2021Backdoor Attacks on Self-Supervised Learningcited by 89
  5. 2021Mean Shift for Self-Supervised Learningcited by 65
  6. 2022Online Continual Learning for Embedded Devicescited by 48
  7. 2022CMD: Self-supervised 3D Action Representation Learning with Cross-modal Mutual Distillationcited by 38
  8. 2022SimA: Simple Softmax-free Attention for Vision Transformerscited by 32
  9. 2022Revisiting Label Smoothing and Knowledge Distillation Compatibility: What was Missing?cited by 32
  10. 2022A Closer Look at Self-Supervised Lightweight Vision Transformerscited by 31
  11. 2024Federated Graph Semantic and Structural Learningcited by 30
  12. 2021DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learningcited by 26
38 more
  1. 2023A Machine Learning-oriented Survey on Tiny Machine Learningcited by 26
  2. 2021Bag of Instances Aggregation Boosts Self-supervised Distillationcited by 22
  3. 2022SimReg: Regression as a Simple Yet Effective Tool for Self-supervised Knowledge Distillationcited by 21
  4. 2023Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learningcited by 20
  5. 2023Defending Against Patch-based Backdoor Attacks on Self-Supervised Learningcited by 19
  6. 2023COMEDIAN: Self-Supervised Learning and Knowledge Distillation for Action Spotting using Transformerscited by 17
  7. 2022MimCo: Masked Image Modeling Pre-training with Contrastive Teachercited by 16
  8. 2022Attention Distillation: self-supervised vision transformer students need more guidancecited by 16
  9. 2021A Simple Baseline for Low-Budget Active Learningcited by 15
  10. 2021On the Efficacy of Small Self-Supervised Contrastive Models without Distillation Signalscited by 14
  11. 2022Distantly-Supervised Named Entity Recognition with Adaptive Teacher Learning and Fine-grained Student Ensemblecited by 14
  12. 2023Learning Common Rationale to Improve Self-Supervised Representation for Fine-Grained Visual Recognition Problemscited by 14
  13. 2022Generalized Knowledge Distillation via Relationship Matchingcited by 12
  14. 2023A Simple Recipe for Competitive Low-compute Self supervised Vision Modelscited by 10
  15. 2022ESTAS: Effective and Stable Trojan Attacks in Self-supervised Encoders with One Target Unlabelled Samplecited by 9
  16. 2024HiMTM: Hierarchical Multi-Scale Masked Time Series Modeling with Self-Distillation for Long-Term Forecastingcited by 8
  17. 2021Boosting Contrastive Learning with Relation Knowledge Distillationcited by 7
  18. 2021Constrained Mean Shift Using Distant Yet Related Neighbors for Representation Learningcited by 7
  19. 2024Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-Training of Deep Networkscited by 6
  20. 2025Backdooring Self-Supervised Contrastive Learning by Noisy Alignmentcited by 6
  21. 2021Simple Distillation Baselines for Improving Small Self-supervised Modelscited by 5
  22. 2022Pixel-Wise Contrastive Distillationcited by 5
  23. 2024Simple Unsupervised Knowledge Distillation With Space Similaritycited by 5
  24. 2022Slimmable Networks for Contrastive Self-supervised Learningcited by 4
  25. 2023Embrace Limited and Imperfect Training Datasets: Opportunities and Challenges in Plant Disease Recognition Using Deep Learningcited by 4
  26. 2024An Experimental Study on Exploring Strong Lightweight Vision Transformers via Masked Image Modeling Pre-Trainingcited by 4
  27. 2024To Distill or Not to Distill? On the Robustness of Robust Knowledge Distillationcited by 4
  28. 2021Towards Communication-Efficient and Privacy-Preserving Federated Representation Learningcited by 3
  29. 2021Auxiliary Learning for Self-Supervised Video Representation via Similarity-based Knowledge Distillationcited by 3
  30. 2022Weak Augmentation Guided Relational Self-Supervised Learningcited by 3
  31. 2022Effective Self-supervised Pre-training on Low-compute Networks without Distillationcited by 3
  32. 2024On Good Practices for Task-Specific Distillation of Large Pretrained Visual Modelscited by 3
  33. 2024Visually Grounded Speech Models have a Mutual Exclusivity Biascited by 3
  34. 2022Similarity Contrastive Estimation for Image and Video Soft Contrastive Self-Supervised Learningcited by 2
  35. 2025Learning Task-Agnostic Representations through Multi-Teacher Distillationcited by 2
  36. 2021Unsupervised Representation Learning for Binary Networks by Joint Classifier Learningcited by 1
  37. 2023I2MD: 3D Action Representation Learning with Inter- and Intra-modal Mutual Distillationcited by 1
  38. 2024Deep Clustering with Diffused Sampling and Hardness-aware Self-distillationcited by 1

50 most cited of 61 All 61 (JSON)