mlsanityData to 7 Oct 2026

Pixel-Wise Contrastive Distillation

arXiv 2211.00218 ↗First posted 1 Nov 2022cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

We present a simple but effective pixel-level self-supervised distillation framework friendly to dense prediction tasks. Our method, called Pixel-Wise Contrastive Distillation (PCD), distills knowledge by attracting the corresponding pixels from student's and teacher's output feature maps. PCD includes a novel design called SpatialAdaptor which ``reshapes'' a part of the teacher network while preserving the distribution of its output features. Our ablation experiments suggest that this reshaping behavior enables more informative pixel-to-pixel distillation. Moreover, we utilize a plug-in multi-head self-attention module that explicitly relates the pixels of student's feature maps to enhance the effective receptive field, leading to a more competitive student. PCD outperforms previous self-supervised distillation methods on various dense prediction tasks. A backbone of mboxResNet-18-FPN distilled by PCD achieves 37.4 APbbox and 34.0 APmask on COCO dataset using the detector of mboxMask R-CNN. We hope our study will inspire future research on how to pre-train a small model friendly to dense prediction tasks in a self-supervised fashion.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 59 of its 69 references that are in the map. First posted on arXiv 1 Nov 2022.

Builds on

59 of 69 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  12. 2017Mask R-CNNcited by 7,259
38 more
  1. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  2. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  4. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  5. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  6. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  7. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  8. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  9. 2019Searching for MobileNetV3cited by 1,984
  10. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  11. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  12. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  13. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  14. 2019Contrastive Multiview Codingcited by 1,297
  15. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  16. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  17. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  18. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  19. 2017Understanding the Effective Receptive Field in Deep Convolutional Neural Networkscited by 549
  20. 2017Large Batch Training of Convolutional Networkscited by 529
  21. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  22. 2013Do Deep Nets Really Need to be Deep?cited by 466
  23. 2019A Comprehensive Overhaul of Feature Distillationcited by 341
  24. 2020Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learningcited by 297
  25. 2021Distilling Knowledge via Knowledge Reviewcited by 285
  26. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 255
  27. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 232
  28. 2021Efficient Visual Pretraining with Contrastive Detectioncited by 150
  29. 2020Unsupervised Learning of Dense Visual Representationscited by 147
  30. 2019Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolutioncited by 140
  31. 2021SEED: Self-supervised Distillation For Visual Representationcited by 132
  32. 2021Aligning Pretraining for Detection via Object-Level Contrastive Learningcited by 110
  33. 2021Instance Localization for Self-supervised Detection Pretrainingcited by 99
  34. 2021Region Similarity Representation Learningcited by 96
  35. 2018Boosting Self-Supervised Learning via Knowledge Transfercited by 92
  36. 2019ClusterFit: Improving Generalization of Visual Representationscited by 82
  37. 2021Unsupervised Object-Level Representation Learning from Scene Imagescited by 72
  38. 2020Self-EMD: Self-Supervised Object Detection without ImageNetcited by 70

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 2.

5 papers.

  1. 2024Aligning in a Compact Space: Contrastive Knowledge Distillation between Heterogeneous Architecturescited by 5
  2. 2024Simple Unsupervised Knowledge Distillation With Space Similaritycited by 5
  3. 2025DNAD: Differentiable Neural Architecture Distillationcited by 0
  4. 2026Training a Student Expert via Semi-Supervised Foundation Model Distillationcited by 0
  5. 2026Weak-to-Strong Knowledge Distillation Accelerates Visual Learningcited by 0