KDC-MAE: Knowledge Distilled Contrastive Mask Auto-Encoder
On the map: Knowledge distillation → General knowledge distillation methods
Abstract (arXiv)
In this work, we attempted to extend the thought and showcase a way forward for the Self-supervised Learning (SSL) learning paradigm by combining contrastive learning, self-distillation (knowledge distillation) and masked data modelling, the three major SSL frameworks, to learn a joint and coordinated representation. The proposed technique of SSL learns by the collaborative power of different learning objectives of SSL. Hence to jointly learn the different SSL objectives we proposed a new SSL architecture KDC-MAE, a complementary masking strategy to learn the modular correspondence, and a weighted way to combine them coordinately. Experimental results conclude that the contrastive masking correspondence along with the KD learning objective has lent a hand to performing better learning for multiple modalities over multiple tasks.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 55 of its 68 references that are in the map. First posted on arXiv 19 Nov 2024.
Builds on
55 of 68 references.
References · 55
- 2017Attention Is All You Needcited by 60,361
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
- 2017The Kinetics Human Action Video Datasetcited by 1,909
- 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
- 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
- 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
- 2017Deep Mutual Learningcited by 628
38 more
- 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
- 2020A Survey on Contrastive Self-supervised Learningcited by 554
- 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
- 2019Domain Generalization by Solving Jigsaw Puzzlescited by 450
- 2015Unsupervised Learning of Video Representations using LSTMscited by 435
- 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
- 2016Generating Videos with Scene Dynamicscited by 376
- 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
- 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 316
- 2020Contrastive learning of global and local features for medical image segmentation with limited annotationscited by 309
- 2016Learning Representations for Automatic Colorizationcited by 251
- 2020Self-supervised Co-training for Video Representation Learningcited by 232
- 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
- 2017Colorization as a Proxy Task for Visual Understandingcited by 175
- 2021BEVT: BERT Pretraining of Video Transformerscited by 169
- 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
- 2022Contrastive Audio-Visual Masked Autoencodercited by 146
- 2018Boosting Self-Supervised Learning via Knowledge Transfercited by 92
- 2020PyTorch Distributed: Experiences on Accelerating Data Parallel Trainingcited by 90
- 2022How Mask Matters: Towards Theoretical Understandings of Masked Autoencoderscited by 86
- 2022Self-Supervised Learning for Videos: A Surveycited by 85
- 2018Learning Image Representations by Completing Damaged Jigsaw Puzzlescited by 66
- 2020SeCo: Exploring Sequence Supervision for Unsupervised Representation Learningcited by 65
- 2022Audio Self-supervised Learning: A Surveycited by 65
- 2022Video Anomaly Detection by Solving Decoupled Spatio-Temporal Jigsaw Puzzlescited by 62
- 2022A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyondcited by 58
- 2022SdAE: Self-distillated Masked Autoencodercited by 52
- 2018Iterative Reorganization with Weak Spatial Constraints: Solving Arbitrary Jigsaw Puzzles for Unsupervised Representation Learningcited by 51
- 2022Audiovisual Masked Autoencoderscited by 49
- 2020Self-supervised Video Representation Learning Using Inter-intra Contrastive Frameworkcited by 46
- 2020Self-Supervised Learning Across Domainscited by 44
- 2023MGMAE: Motion Guided Masking for Video Masked Autoencodingcited by 41
- 2020Enhancing Unsupervised Video Representation Learning by Decoupling the Scene and the Motioncited by 39
- 2022How to Understand Masked Autoencoderscited by 38
- 2022Masked Jigsaw Puzzle: A Versatile Position Embedding for Vision Transformerscited by 33
- 2020Puzzle-AE: Novelty Detection in Images through Solving Puzzlescited by 23
- 2022Self-supervised Video Representation Learning with Motion-Aware Masked Autoencoderscited by 20
- 2021Similarity Contrastive Estimation for Self-Supervised Soft Contrastive Learningcited by 18
50 most cited of 55 All 55 (JSON)
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023MOMA:Distill from Self-Supervised Teachers
- 2021Self-Distilled Self-Supervised Representation Learning
- 2023Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners
- 2023Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learning
- 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining
- 2021Boosting Contrastive Learning with Relation Knowledge Distillation
- 2024Frequency-Guided Masking for Enhanced Vision Self-Supervised Learning
- 2022Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation