mlsanityData to 7 Oct 2026

Efficiently Distilling LLMs for Edge Applications

arXiv 2404.01353 ↗First posted 1 Apr 2024cs.LG cs.AI cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Supernet training of LLMs is of great interest in industrial applications as it confers the ability to produce a palette of smaller models at constant cost, regardless of the number of models (of different size / latency) produced. We propose a new method called Multistage Low-rank Fine-tuning of Super-transformers (MLFS) for parameter-efficient supernet training. We show that it is possible to obtain high-quality encoder models that are suitable for commercial edge applications, and that while decoder-only models are resistant to a comparable degree of compression, decoders can be effectively sliced for a significant reduction in training time.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 43 of its 51 references that are in the map. First posted on arXiv 1 Apr 2024.

Builds on

43 of 51 references.

References · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  10. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  12. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
31 more
  1. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  2. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  3. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  4. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  5. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  6. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 411
  7. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  8. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  9. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  10. 2023A Survey on Model Compression for Large Language Modelscited by 263
  11. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  12. 2020BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Modelscited by 187
  13. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 186
  14. 2023SantaCoder: don't reach for the stars!cited by 169
  15. 2020HAT: Hardware-Aware Transformers for Efficient Natural Language Processingcited by 136
  16. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  17. 2019A Tensorized Transformer for Language Modelingcited by 85
  18. 2020AttentiveNAS: Improving Neural Architecture Search via Attentive Samplingcited by 65
  19. 2019Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Modelcited by 58
  20. 2021NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Searchcited by 55
  21. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 45
  22. 2020XtremeDistil: Multi-stage Distillation for Massive Multilingual Modelscited by 38
  23. 2023Weight subcloning: direct initialization of transformers using larger pretrained onescited by 30
  24. 2021AutoBERT-Zero: Evolving BERT Backbone from Scratchcited by 21
  25. 2021Dynamic-OFA: Runtime DNN Architecture Switching for Performance Scaling on Heterogeneous Embedded Platformscited by 18
  26. 2021XtremeDistilTransformers: Task Transfer for Task-agnostic Distillationcited by 18
  27. 2021You Only Compress Once: Towards Effective and Elastic BERT Compression via Exploit-Explore Stochastic Nature Gradientcited by 14
  28. 2022Prior-Guided One-shot Neural Architecture Searchcited by 12
  29. 2019EENA: Efficient Evolution of Neural Architecturecited by 10
  30. 2023Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with Architecture-Routed Mixture-of-Expertscited by 7
  31. 2023Transfer-Once-For-All: AI Model Optimization for Edgecited by 4

Built on by

Cited by year

Citing papers per year: 2024: 4; 2025: 1.

5 papers.

  1. 2024CE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaborationcited by 25
  2. 2024Empirical Guidelines for Deploying LLMs onto Resource-constrained Edge Devicescited by 17
  3. 2024New Solutions on LLM Acceleration, Optimization, and Applicationcited by 7
  4. 2025T-SAR: A Full-Stack Co-design for CPU-Only Ternary LLM Inference via In-Place SIMD ALU Reorganizationcited by 4
  5. 2024SoftmAP: Software-Hardware Co-design for Integer-Only Softmax on Associative Processorscited by 3