mlsanityData to 14 Oct 2026

Arch-Net: Model Distillation for Architecture Agnostic Model Deployment

arXiv 2111.01135 ↗First posted 1 Nov 2021cs.LG cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

Vast requirement of computation power of Deep Neural Networks is a major hurdle to their real world applications. Many recent Application Specific Integrated Circuit (ASIC) chips feature dedicated hardware support for Neural Network Acceleration. However, as ASICs take multiple years to develop, they are inevitably out-paced by the latest development in Neural Architecture Research. For example, Transformer Networks do not have native support on many popular chips, and hence are difficult to deploy. In this paper, we propose Arch-Net, a family of Neural Networks made up of only operators efficiently supported across most architectures of ASICs. When a Arch-Net is produced, less common network constructs, like Layer Normalization and Embedding Layers, are eliminated in a progressive manner through label-free Blockwise Model Distillation, while performing sub-eight bit quantization at the same time to maximize performance. Empirical results on machine translation and image classification tasks confirm that we can transform latest developed Neural Architectures into fast running and as-accurate Arch-Net, ready for deployment on multiple mass-produced ASIC chips. The code will be available at https://github.com/megvii-research/Arch-Net.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 48 of its 54 references that are in the map. First posted on arXiv 1 Nov 2021.

Builds on

48 of 54 references.

References · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  8. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  9. 2017Squeeze-and-Excitation Networkscited by 4,789
  10. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  11. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  12. 2014FitNets: Hints for Thin Deep Netscited by 1,653
36 more
  1. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  2. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  3. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  4. 2019Relational Knowledge Distillationcited by 889
  5. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  6. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  7. 2017Deep Mutual Learningcited by 628
  8. 2019Learned Step Size Quantizationcited by 628
  9. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  10. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  11. 2021RepVGG: Making VGG-style ConvNets Great Againcited by 522
  12. 2019Similarity-Preserving Knowledge Distillationcited by 516
  13. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  14. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  15. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
  16. 2019On the Efficacy of Knowledge Distillationcited by 381
  17. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 336
  18. 2018Model compression via distillation and quantizationcited by 289
  19. 2019Structured Knowledge Distillation for Dense Predictioncited by 279
  20. 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 261
  21. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  22. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 238
  23. 2016Trained Ternary Quantizationcited by 208
  24. 2019Improving Neural Network Quantization without Retraining using Outlier Channel Splittingcited by 151
  25. 2017Deep Learning with Low Precision by Half-wave Gaussian Quantizationcited by 117
  26. 2016Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1cited by 105
  27. 2017Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracycited by 101
  28. 2020Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programmingcited by 92
  29. 2020Post-Training Piecewise Linear Quantization for Deep Neural Networkscited by 79
  30. 2016Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computingcited by 70
  31. 2021Diversifying Sample Generation for Accurate Data-Free Quantizationcited by 63
  32. 2018MEAL: Multi-Model Ensemble via Adversarial Learningcited by 52
  33. 2019QKD: Quantization-aware Knowledge Distillationcited by 44
  34. 2018Post-training 4-bit quantization of convolution networks for rapid-deploymentcited by 42
  35. 2020MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Trickscited by 40
  36. 2018Learning Sparse Low-Precision Neural Networks With Learnable Regularizationcited by 4

Built on by

Cited by year

Citing papers per year: 2022: 1.

1 paper.

  1. 2022One is All: Bridging the Gap Between Neural Radiance Fields Architectures with Progressive Volume Distillationcited by 18

Similar, not cited

Close in wording, no citation either way

  1. 2024From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks
  2. 2023ArchBERT: Bi-Modal Understanding of Neural Architectures and Natural Languages
  3. 2021Compact CNN Structure Learning by Knowledge Distillation
  4. 2021Pre-Quantized Deep Learning Models Codified in ONNX to Enable Hardware/Software Co-Design
  5. 2025Hardware-Aware Neural Network Compilation with Learned Optimization: A RISC-V Accelerator Approach
  6. 2025Low-bit Model Quantization for Deep Neural Networks: A Survey
  7. 2025On Accelerating Edge AI: Optimizing Resource-Constrained Environments
  8. 2022SpeedLimit: Neural Architecture Search for Quantized Transformer Models