mlsanityData to 14 Oct 2026

Self-Distillation of Hidden Layers for Self-Supervised Representation Learning

arXiv 2603.15553 ↗First posted 16 Mar 2026cs.CV cs.LGcited by 3

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

The landscape of self-supervised learning (SSL) is currently dominated by generative approaches (e.g. MAE) that reconstruct raw low-level data, and predictive approaches (e.g. I-JEPA) that predict high-level abstract embeddings. While generative methods are stable due to their reliable training targets based on ground-truth data, they are computationally inefficient for high-redundancy modalities like imagery, and their training objective does not prioritize learning high-level, conceptual features. Conversely, predictive methods often suffer from training instability due to their reliance on the non-stationary targets of final-layer self-distillation. We introduce Bootleg, a method that bridges this divide by tasking the model with predicting latent representations from multiple hidden layers of a teacher. This hierarchical objective forces the model to capture features at varying levels of abstraction simultaneously. We demonstrate Bootleg significantly outperforms comparable baselines (+10% vs. I-JEPA) on frozen probe classification of ImageNet-1K, iNaturalist-21, and VTAB, and semantic segmentation of ADE20K, Cityscapes, and COCO-Stuff.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 65 of its 83 references that are in the map. First posted on arXiv 16 Mar 2026.

Builds on

65 of 83 references.

References · 65

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  11. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  12. 2020Exploring Simple Siamese Representation Learningcited by 2,866
38 more
  1. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  2. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  3. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  4. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  5. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
  6. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  7. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  8. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  9. 2025DINOv3cited by 1,414
  10. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  11. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  12. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  13. 2023Vision Transformers Need Registerscited by 871
  14. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  15. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  16. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  17. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 708
  18. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  19. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  20. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
  21. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  22. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  23. 2022SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagerycited by 408
  24. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  25. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 291
  26. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 264
  27. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  28. 2014Object Detectors Emerge in Deep Scene CNNscited by 179
  29. 2015Generic decoding of seen and imagined objects using hierarchical visual featurescited by 120
  30. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 110
  31. 2023What Do Self-Supervised Vision Transformers Learn?cited by 93
  32. 2022Head2Toe: Utilizing Intermediate Representations for Better Transfer Learningcited by 83
  33. 2022Self Pre-training with Masked Autoencoders for Medical Image Classification and Segmentationcited by 69
  34. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 68
  35. 2022Delving into Masked Autoencoders for Multi-Label Thorax Disease Classificationcited by 65
  36. 2024Brain-JEPA: Brain Dynamics Foundation Model with Gradient Positioning and Spatiotemporal Maskingcited by 65
  37. 2023Ti-MAE: Self-Supervised Masked Time Series Autoencoderscited by 63
  38. 2023PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detectioncited by 53

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Learn from your own latents and not from tokens: A sample-complexity theorycited by 2
  2. 2026I Have a Stream: Making Self-Supervised Learning Work on Continuous Videocited by 0
  3. 2026Scalable Patch-Level Self-Supervised Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Adaptive Similarity Bootstrapping for Self-Distillation based Representation Learning
  2. 2021Self-Distilled Self-Supervised Representation Learning
  3. 2026Self-Supervised Representation-Guided Generative Dataset Distillation
  4. 2023DreamTeacher: Pretraining Image Backbones with Deep Generative Models
  5. 2022Exploring Target Representations for Masked Autoencoders
  6. 2023A Simple Recipe for Competitive Low-compute Self supervised Vision Models
  7. 2021Hierarchical Self-supervised Augmented Knowledge Distillation
  8. 2021Hyperspherically Regularized Networks for Self-Supervision