mlsanityData to 14 Oct 2026

GhostNetV3-Small: A Tailored Architecture and Comparative Study of Distillation Strategies for Tiny Images

arXiv 2509.12380 ↗First posted 15 Sep 2025cs.CV cs.AI cs.LGcited by 0

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Deep neural networks have achieved remarkable success across a range of tasks, however their computational demands often make them unsuitable for deployment on resource-constrained edge devices. This paper explores strategies for compressing and adapting models to enable efficient inference in such environments. We focus on GhostNetV3, a state-of-the-art architecture for mobile applications, and propose GhostNetV3-Small, a modified variant designed to perform better on low-resolution inputs such as those in the CIFAR-10 dataset. In addition to architectural adaptation, we provide a comparative evaluation of knowledge distillation techniques, including traditional knowledge distillation, teacher assistants, and teacher ensembles. Experimental results show that GhostNetV3-Small significantly outperforms the original GhostNetV3 on CIFAR-10, achieving an accuracy of 93.94%. Contrary to expectations, all examined distillation strategies led to reduced accuracy compared to baseline training. These findings indicate that architectural adaptation can be more impactful than distillation in small-scale image classification tasks, highlighting the need for further research on effective model design and advanced distillation techniques for low-resolution domains.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 23 of its 31 references that are in the map. First posted on arXiv 15 Sep 2025.

Builds on

23 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Adaptive Multi-Teacher Multi-level Knowledge Distillation6 Mar

Other references · 22

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2016Densely Connected Convolutional Networkscited by 6,720
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  8. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  9. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  10. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
  11. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  12. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
10 more
  1. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  2. 2019GhostNet: More Features from Cheap Operationscited by 410
  3. 2019On the Efficacy of Knowledge Distillationcited by 381
  4. 2017NISP: Pruning Networks using Neuron Importance Score Propagationcited by 228
  5. 2019Online Knowledge Distillation with Diverse Peerscited by 128
  6. 2020Densely Guided Knowledge Distillation using Multiple Teacher Assistantscited by 83
  7. 2018MEAL: Multi-Model Ensemble via Adversarial Learningcited by 52
  8. 2022GhostNetV2: Enhance Cheap Operation with Long-Range Attentioncited by 44
  9. 2024GhostNetV3: Exploring the Training Strategies for Compact Modelscited by 14
  10. 2017The CAPIO 2017 Conversational Speech Recognition Systemcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Systematic Characterization of Minimal Deep Learning Architectures: A Unified Analysis of Convergence, Pruning, and Quantization
  2. 2021Compact CNN Structure Learning by Knowledge Distillation
  3. 2023A Simple Recipe for Competitive Low-compute Self supervised Vision Models
  4. 2026Efficient Few-Shot Learning for Edge AI via Knowledge Distillation on MobileViT
  5. 2023TinyMIM: An Empirical Study of Distilling MIM Pre-trained Models
  6. 2021Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve Backbones
  7. 2024Tiny models from tiny data: Textual and null-text inversion for few-shot distillation
  8. 2025Distillation-Supervised Convolutional Low-Rank Adaptation for Efficient Image Super-Resolution