TinySSL: Distilled Self-Supervised Pretraining for Sub-Megabyte MCU Models
On the map: Knowledge distillation → General knowledge distillation methods
Abstract (arXiv)
Self-supervised learning (SSL) has transformed representation learning for large models, yet remains unexplored for microcontroller (MCU)-class models with fewer than 500K parameters. We identify three obstacles at this scale -- projection head dominance, representation bottleneck, and augmentation sensitivity -- and propose Capacity-Aware Distilled Self-Supervised Learning (CA-DSSL), a teacher-guided framework that overcomes them without labels or text supervision. CA-DSSL combines asymmetric distillation from a frozen DINO ViT-S/16 teacher, multi-scale feature distillation for spatial representations, and a progressive augmentation curriculum. On a MobileNetV2-0.35 backbone (396K parameters) pretrained on CIFAR-100, CA-DSSL reaches 62.7 0.5% linear-probe accuracy (3-seed mean) -- surpassing SimCLR-Tiny by 18 pp, matching SEED (61.7%) with 10 fewer projection parameters (426K vs. 3.15M), and reaching 94.0% of a supervised upper bound. Standard SSL methods (BYOL-Tiny, DINO-Tiny) collapse entirely at this scale. On Pascal VOC detection, CA-DSSL achieves 2.3 the mAP of random initialization and +3 pp over SEED, though SimCLR-Tiny matches CA-DSSL on detection mAP. The deployed backbone occupies 378 KB (INT8) with no inference overhead from pretraining. Preliminary ImageNet-100 experiments reveal that CA-DSSL's advantage is specific to small-data regimes; scaling to ImageNet-1K is discussed as future work.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 17 of its 19 references that are in the map. First posted on arXiv 7 May 2026.
Builds on
17 of 19 references.
References · 17
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
- 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
- 2020Exploring Simple Siamese Representation Learningcited by 2,866
- 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
- 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
- 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,577
- 2020MCUNet: Tiny Deep Learning on IoT Devicescited by 273
5 more
- 2021SEED: Self-supervised Distillation For Visual Representationcited by 134
- 2021MLPerf Tiny Benchmarkcited by 124
- 2021MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learningcited by 89
- 2020CompRess: Self-Supervised Learning by Compressing Representationscited by 62
- 2021DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learningcited by 26
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023A Simple Recipe for Competitive Low-compute Self supervised Vision Models
- 2022Effective Self-supervised Pre-training on Low-compute Networks without Distillation
- 2021Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve Backbones
- 2023MOMA:Distill from Self-Supervised Teachers
- 2023DMT: Comprehensive Distillation with Multiple Self-supervised Teachers
- 2023TinyMIM: An Empirical Study of Distilling MIM Pre-trained Models
- 2022SimReg: Regression as a Simple Yet Effective Tool for Self-supervised Knowledge Distillation
- 2025A Layered Self-Supervised Knowledge Distillation Framework for Efficient Multimodal Learning on the Edge