mlsanityData to 14 Oct 2026

SOLAR: Switchable Output Layer for Accuracy and Robustness in Once-for-All Training

arXiv 2509.16833 ↗First posted 20 Sep 2025cs.LG cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Once-for-All (OFA) training enables a single super-net to generate multiple sub-nets tailored to diverse deployment scenarios, supporting flexible trade-offs among accuracy, robustness, and model-size without retraining. However, as the number of supported sub-nets increases, excessive parameter sharing in the backbone limits representational capacity, leading to degraded calibration and reduced overall performance. To address this, we propose SOLAR (Switchable Output Layer for Accuracy and Robustness in Once-for-All Training), a simple yet effective technique that assigns each sub-net a separate classification head. By decoupling the logit learning process across sub-nets, the Switchable Output Layer (SOL) reduces representational interference and improves optimization, without altering the shared backbone. We evaluate SOLAR on five datasets (SVHN, CIFAR-10, STL-10, CIFAR-100, and TinyImageNet) using four super-net backbones (ResNet-34, WideResNet-16-8, WideResNet-40-2, and MobileNetV2) for two OFA training frameworks (OATS and SNNs). Experiments show that SOLAR outperforms the baseline methods: compared to OATS, it improves accuracy of sub-nets up to 1.26 %, 4.71 %, 1.67 %, and 1.76 %, and robustness up to 9.01 %, 7.71 %, 2.72 %, and 1.26 % on SVHN, CIFAR-10, STL-10, and CIFAR-100, respectively. Compared to SNNs, it improves TinyImageNet accuracy by up to 2.93 %, 2.34 %, and 1.35 % using ResNet-34, WideResNet-16-8, and MobileNetV2 backbones (with 8 sub-nets), respectively.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 36 of its 50 references that are in the map. First posted on arXiv 20 Sep 2025.

Builds on

36 of 50 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  7. 2016Wide Residual Networkscited by 3,070
  8. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  9. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  10. 2020Knowledge Distillation: A Surveycited by 1,603
  11. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  12. 2018Robustness May Be at Odds with Accuracycited by 880
24 more
  1. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  2. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 776
  3. 2019Adversarial Training for Free!cited by 643
  4. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 332
  5. 2018Slimmable Neural Networkscited by 305
  6. 2021Recent Advances in Adversarial Training for Adversarial Robustnesscited by 279
  7. 2019Universally Slimmable Networks and Improved Training Techniquescited by 221
  8. 2018MMA Training: Direct Input Space Margin Maximization through Adversarial Trainingcited by 172
  9. 2018Understanding Batch Normalizationcited by 159
  10. 2020Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuningcited by 153
  11. 2019Adversarially Robust Distillationcited by 125
  12. 2024Logit Standardization in Knowledge Distillationcited by 102
  13. 2018Curriculum Adversarial Trainingcited by 89
  14. 2021Only Train Once: A One-Shot Neural Network Training And Pruning Frameworkcited by 70
  15. 2020Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Freecited by 48
  16. 2021Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neuronscited by 40
  17. 2019Batch Normalization is a Cause of Adversarial Vulnerabilitycited by 29
  18. 2022Removing Batch Normalization Boosts Adversarial Trainingcited by 24
  19. 2019On the Sensitivity of Adversarial Robustness to Input Data Distributionscited by 20
  20. 2020Bridging the Performance Gap between FGSM and PGD Adversarial Trainingcited by 9
  21. 2023Three Guidelines You Should Know for Universally Slimmable Self-Supervised Learningcited by 5
  22. 2022Slimmable Networks for Contrastive Self-supervised Learningcited by 4
  23. 2022Sparse Mixture Once-for-all Adversarial Training for Efficient In-Situ Trade-Off Between Accuracy and Robustness of DNNscited by 4
  24. 2023Progressive Channel-Shrinking Networkcited by 4

Built on by

Not cited yet.