mlsanityData to 14 Oct 2026

LANA: Latency Aware Network Acceleration

arXiv 2107.10624 ↗First posted 12 Jul 2021cs.CV cs.AI cs.LGcited by 11

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

We introduce latency-aware network acceleration (LANA) - an approach that builds on neural architecture search techniques and teacher-student distillation to accelerate neural networks. LANA consists of two phases: in the first phase, it trains many alternative operations for every layer of the teacher network using layer-wise feature map distillation. In the second phase, it solves the combinatorial selection of efficient operations using a novel constrained integer linear optimization (ILP) approach. ILP brings unique properties as it (i) performs NAS within a few seconds to minutes, (ii) easily satisfies budget constraints, (iii) works on the layer-granularity, (iv) supports a huge search space O(10100), surpassing prior search approaches in efficacy and efficiency. In extensive experiments, we show that LANA yields efficient and accurate models constrained by a target latency budget, while being significantly faster than other techniques. We analyze three popular network architectures: EfficientNetV1, EfficientNetV2 and ResNeST, and achieve accuracy improvement for all models (up to 3.0%) when compressing larger models to the latency level of smaller models. LANA achieves significant speed-ups (up to 5×) with minor to no accuracy drop on GPU and CPU. The code will be shared soon.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 89 of its 106 references that are in the map. First posted on arXiv 12 Jul 2021.

Builds on

89 of 106 references.

References · 89

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2016Densely Connected Convolutional Networkscited by 6,720
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  8. 2017Squeeze-and-Excitation Networkscited by 4,789
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  10. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  11. 2016Wide Residual Networkscited by 3,070
  12. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
38 more
  1. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,332
  2. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  3. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  4. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  5. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,512
  6. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  7. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,420
  8. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  9. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  10. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
  11. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,101
  12. 2020Designing Network Design Spacescited by 960
  13. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 943
  14. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  15. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 837
  16. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  17. 2021Going deeper with Image Transformerscited by 764
  18. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  19. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 673
  20. 2018Rethinking the Value of Network Pruningcited by 599
  21. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  22. 2020What is the State of Neural Network Pruning?cited by 583
  23. 2019ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networkscited by 575
  24. 2019Importance Estimation for Neural Network Pruningcited by 541
  25. 2021RepVGG: Making VGG-style ConvNets Great Againcited by 522
  26. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 521
  27. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 485
  28. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  29. 2020ResNeSt: Split-Attention Networkscited by 445
  30. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 443
  31. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 442
  32. 2017Deep Layer Aggregationcited by 436
  33. 2017Efficient Processing of Deep Neural Networks: A Tutorial and Surveycited by 434
  34. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 431
  35. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 423
  36. 2019Selective Kernel Networkscited by 403
  37. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
  38. 2019CSPNet: A New Backbone that can Enhance Learning Capability of CNNcited by 381

50 most cited of 89 All 89 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 5; 2024: 1; 2025: 3.

11 papers.

  1. 2023FasterViT: Fast Vision Transformers with Hierarchical Attentioncited by 65
  2. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 41
  3. 2025Fast-FoundationStereo: Real-Time Zero-Shot Stereo Matchingcited by 28
  4. 2023Adaptive Sharpness-Aware Pruning for Robust Sparse Networkscited by 17
  5. 2022BLOX: Macro Neural Architecture Search Benchmark and Algorithmscited by 6
  6. 2025MDP: Multidimensional Vision Model Pruning with Latency Constraintcited by 5
  7. 2023Neural Fine-Tuning Search for Few-Shot Learningcited by 4
  8. 2023MathNAS: If Blocks Have a Role in Mathematical Architecture Designcited by 4
  9. 2024Multi-Dimensional Pruning: Joint Channel, Layer and Block Pruning with Latency Constraintcited by 4
  10. 2022Simple and Efficient Architectures for Semantic Segmentationcited by 2
  11. 2023Bespoke: A Block-Level Neural Network Optimization Framework for Low-Cost Deploymentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025On Accelerating Edge AI: Optimizing Resource-Constrained Environments
  2. 2021FOX-NAS: Fast, On-device and Explainable Neural Architecture Search
  3. 2026Latency-Constrained DNN Architecture Learning for Edge Systems using Zerorized Batch Normalization
  4. 2023DONNAv2 -- Lightweight Neural Architecture Search for Vision tasks
  5. 2021CONetV2: Efficient Auto-Channel Size Optimization for CNNs
  6. 2023Improving Differentiable Architecture Search via Self-Distillation
  7. 2025RAM-NAS: Resource-aware Multiobjective Neural Architecture Search Method for Robot Vision Tasks
  8. 2025Layer-wise Weight Selection for Power-Efficient Neural Network Acceleration