mlsanityData to 14 Oct 2026

Auxiliary Learning for Self-Supervised Video Representation via Similarity-based Knowledge Distillation

arXiv 2112.04011 ↗First posted 7 Dec 2021cs.CVcited by 3

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Despite the outstanding success of self-supervised pretraining methods for video representation learning, they generalise poorly when the unlabeled dataset for pretraining is small or the domain difference between unlabelled data in source task (pretraining) and labeled data in target task (finetuning) is significant. To mitigate these issues, we propose a novel approach to complement self-supervised pretraining via an auxiliary pretraining phase, based on knowledge similarity distillation, auxSKD, for better generalisation with a significantly smaller amount of video data, e.g. Kinetics-100 rather than Kinetics-400. Our method deploys a teacher network that iteratively distills its knowledge to the student model by capturing the similarity information between segments of unlabelled video data. The student model meanwhile solves a pretext task by exploiting this prior knowledge. We also introduce a novel pretext task, Video Segment Pace Prediction or VSPP, which requires our model to predict the playback speed of a randomly selected segment of the input video to provide more reliable self-supervised representations. Our experimental results show superior results to the state of the art on both UCF101 and HMDB51 datasets when pretraining on K100 in apple-to-apple comparisons. Additionally, we show that our auxiliary pretraining, auxSKD, when added as an extra pretraining phase to recent state of the art self-supervised methods (i.e. VCOP, VideoPace, and RSPNet), improves their results on UCF101 and HMDB51. Our code is available at https://github.com/Plrbear/auxSKD.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 40 of its 51 references that are in the map. First posted on arXiv 7 Dec 2021.

Builds on

40 of 51 references.

References · 40

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  4. 2017The Kinetics Human Action Video Datasetcited by 1,909
  5. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  6. 2019Contrastive Multiview Codingcited by 1,301
  7. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  8. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  9. 2019Relational Knowledge Distillationcited by 889
  10. 2019Contrastive Representation Distillationcited by 748
  11. 2017Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?cited by 551
  12. 2019Similarity-Preserving Knowledge Distillationcited by 516
28 more
  1. 2019S4L: Self-Supervised Semi-Supervised Learningcited by 335
  2. 2019Correlation Congruence for Knowledge Distillationcited by 285
  3. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 267
  4. 2017Unsupervised Representation Learning by Sorting Sequencescited by 221
  5. 2015Simultaneous Deep Transfer Across Domains and Taskscited by 206
  6. 2020SpeedNet: Learning the Speediness in Videoscited by 171
  7. 2021VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examplescited by 154
  8. 2021Efficient Visual Pretraining with Contrastive Detectioncited by 150
  9. 2018Self-Supervised Video Representation Learning with Space-Time Cubic Puzzlescited by 149
  10. 2020Knowledge Distillation Meets Self-Supervisioncited by 142
  11. 2020Self-supervised Video Representation Learning by Pace Predictioncited by 139
  12. 2021SEED: Self-supervised Distillation For Visual Representationcited by 134
  13. 2020Video Playback Rate Perception for Self-supervisedSpatio-Temporal Representation Learningcited by 103
  14. 2018Boosting Self-Supervised Learning via Knowledge Transfercited by 92
  15. 2019Self-Supervised Generalisation with Meta Auxiliary Learningcited by 88
  16. 2020Video Representation Learning by Recognizing Temporal Transformationscited by 83
  17. 2020RSPNet: Relative Speed Perception for Unsupervised Video Representation Learningcited by 83
  18. 2019Oops! Predicting Unintentional Action in Videocited by 82
  19. 2021Self-Supervised Pretraining Improves Self-Supervised Pretrainingcited by 82
  20. 2020Video Cloze Procedure for Self-Supervised Spatio-Temporal Learningcited by 81
  21. 2020CompRess: Self-Supervised Learning by Compressing Representationscited by 62
  22. 2020Auxiliary Learning by Implicit Differentiationcited by 45
  23. 2021ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistencycited by 35
  24. 2020ISD: Self-Supervised Learning by Iterative Similarity Distillationcited by 33
  25. 2021Self-Supervised Video Representation Learning with Meta-Contrastive Networkcited by 28
  26. 2020Auxiliary Task Reweighting for Minimum-data Learningcited by 23
  27. 2021Unsupervised Visual Representation Learning by Tracking Patches in Videocited by 20
  28. 2020Back to the Future: Cycle Encoding Prediction for Self-supervised Contrastive Video Representation Learningcited by 4

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 0; 2025: 1.

3 papers.

  1. 2023PECoP: Parameter Efficient Continual Pretraining for Action Quality Assessmentcited by 19
  2. 2023XVO: Generalized Visual Odometry via Cross-Modal Self-Trainingcited by 18
  3. 2025PSMamba: Progressive Self-supervised Vision Mamba for Plant Disease Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Preserve Pre-trained Knowledge: Transfer Learning With Self-Distillation For Action Recognition
  2. 2023SELF-VS: Self-supervised Encoding Learning For Video Summarization
  3. 2021Hierarchical Self-supervised Augmented Knowledge Distillation
  4. 2025When the Future Becomes the Past: Taming Temporal Correspondence for Self-supervised Video Representation Learning
  5. 2023The Staged Knowledge Distillation in Video Classification: Harmonizing Student Progress by a Complementary Weakly Supervised Framework
  6. 2022XKD: Cross-modal Knowledge Distillation with Domain Alignment for Video Representation Learning
  7. 2022Distilling Knowledge from Self-Supervised Teacher by Embedding Graph Alignment
  8. 2022Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learning