mlsanityData to 7 Oct 2026

Robustness to distribution shifts of compressed networks for edge devices

arXiv 2401.12014 ↗First posted 22 Jan 2024cs.LG cs.AI cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

It is necessary to develop efficient DNNs deployed on edge devices with limited computation resources. However, the compressed networks often execute new tasks in the target domain, which is different from the source domain where the original network is trained. It is important to investigate the robustness of compressed networks in two types of data distribution shifts: domain shifts and adversarial perturbations. In this study, we discover that compressed models are less robust to distribution shifts than their original networks. Interestingly, larger networks are more vulnerable to losing robustness than smaller ones, even when they are compressed to a similar size as the smaller networks. Furthermore, compact networks obtained by knowledge distillation are much more robust to distribution shifts than pruned networks. Finally, post-training quantization is a reliable method for achieving significant robustness to distribution shifts, and it outperforms both pruned and distilled models in terms of robustness.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 42 of its 52 references that are in the map. First posted on arXiv 22 Jan 2024.

Builds on

42 of 52 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  8. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  9. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  10. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  11. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  12. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
30 more
  1. 2020Knowledge Distillation: A Surveycited by 1,578
  2. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  3. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  4. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  5. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
  6. 2016Practical Black-Box Attacks against Machine Learningcited by 893
  7. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  8. 2018Deep Visual Domain Adaptation: A Surveycited by 634
  9. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  10. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 542
  11. 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 365
  12. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
  13. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 298
  14. 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 291
  15. 2019Towards Stable and Efficient Training of Verifiably Robust Neural Networkscited by 204
  16. 2020Binary Neural Networks: A Surveycited by 184
  17. 2019Adversarial Robustness through Local Linearizationcited by 146
  18. 2018Decoupling Direction and Norm for Efficient Gradient-Based L2 Adversarial Attacks and Defensescited by 109
  19. 2019Adversarial Robustness vs Model Compression, or Both?cited by 98
  20. 2019Defensive Quantization: When Efficiency Meets Robustnesscited by 95
  21. 2019AdderNet: Do We Really Need Multiplications in Deep Learning?cited by 82
  22. 2018Sparse DNNs with Improved Adversarial Robustnesscited by 71
  23. 2019Model Compression with Adversarial Robustness: A Unified Optimization Frameworkcited by 71
  24. 2021How Well Do Sparse Imagenet Models Transfer?cited by 50
  25. 2020Pruning Filter in Filtercited by 46
  26. 2020A Survey on Deep Neural Network Compression: Challenges, Overview, and Solutionscited by 32
  27. 2021On the Effect of Pruning on Adversarial Robustnesscited by 23
  28. 2018Filter Distillation for Network Compressioncited by 17
  29. 2018To compress or not to compress: Understanding the Interactions between Adversarial Attacks and Neural Network Compressioncited by 14
  30. 2020Adversarial Feature Desensitizationcited by 12

Built on by

Not cited yet.