mlsanityData to 14 Oct 2026

OVO: One-shot Vision Transformer Search with Online distillation

arXiv 2212.13766 ↗First posted 28 Dec 2022cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Pure transformers have shown great potential for vision tasks recently. However, their accuracy in small or medium datasets is not satisfactory. Although some existing methods introduce a CNN as a teacher to guide the training process by distillation, the gap between teacher and student networks would lead to sub-optimal performance. In this work, we propose a new One-shot Vision transformer search framework with Online distillation, namely OVO. OVO samples sub-nets for both teacher and student networks for better distillation results. Benefiting from the online distillation, thousands of subnets in the supernet are well-trained without extra finetuning or retraining. In experiments, OVO-Ti achieves 73.32% top-1 accuracy on ImageNet and 75.2% on CIFAR-100, respectively.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 39 references that are in the map. First posted on arXiv 28 Dec 2022.

Builds on

33 of 39 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  6. 2016Layer Normalizationcited by 5,669
  7. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  10. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  12. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
21 more
  1. 2019Searching for MobileNetV3cited by 1,994
  2. 2020Knowledge Distillation: A Surveycited by 1,603
  3. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  4. 2021Transformers in Vision: A Surveycited by 1,013
  5. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 943
  6. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  7. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
  8. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 837
  9. 2019Similarity-Preserving Knowledge Distillationcited by 516
  10. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 485
  11. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 414
  12. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  13. 2018Slimmable Neural Networkscited by 305
  14. 2019Random Search and Reproducibility for Neural Architecture Searchcited by 254
  15. 2020BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Modelscited by 188
  16. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 188
  17. 2019FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Searchcited by 130
  18. 2019Knowledge Adaptation for Efficient Semantic Segmentationcited by 105
  19. 2020Heterogeneous Knowledge Distillation using Information Flow Modelingcited by 73
  20. 2020Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Searchcited by 38
  21. 2022Locality Guidance for Improving Vision Transformers on Tiny Datasetscited by 33

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023TVT: Training-Free Vision Transformer Search on Tiny Datasets
  2. 2024Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision Transformers
  3. 2021Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation
  4. 2022DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformers
  5. 2023A survey on efficient vision transformers: algorithms, techniques, and performance benchmarking
  6. 2024AMD: Automatic Multi-step Distillation of Large-scale Vision Models
  7. 2025Image Recognition with Online Lightweight Vision Transformer: A Survey
  8. 2022Attention Distillation: self-supervised vision transformer students need more guidance