mlsanityData to 7 Oct 2026

AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models

arXiv 2107.13686 ↗First posted 29 Jul 2021cs.CL cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models (PLMs) have achieved great success in natural language processing. Most of PLMs follow the default setting of architecture hyper-parameters (e.g., the hidden dimension is a quarter of the intermediate dimension in feed-forward sub-networks) in BERT (Devlin et al., 2019). Few studies have been conducted to explore the design of architecture hyper-parameters in BERT, especially for the more efficient PLMs with tiny sizes, which are essential for practical deployment on resource-constrained devices. In this paper, we adopt the one-shot Neural Architecture Search (NAS) to automatically search architecture hyper-parameters. Specifically, we carefully design the techniques of one-shot learning and the search space to provide an adaptive and efficient development way of tiny PLMs for various latency constraints. We name our method AutoTinyBERT and evaluate its effectiveness on the GLUE and SQuAD benchmarks. The extensive experiments show that our method outperforms both the SOTA search-based baseline (NAS-BERT) and the SOTA distillation-based methods (such as DistilBERT, TinyBERT, MiniLM and MobileBERT). In addition, based on the obtained architectures, we propose a more efficient development method that is even faster than the development of a single PLM.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 36 of its 41 references that are in the map. First posted on arXiv 29 Jul 2021.

Builds on

36 of 41 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
24 more
  1. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  2. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  3. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  4. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  5. 2019EfficientDet: Scalable and Efficient Object Detectioncited by 1,018
  6. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  7. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  8. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  9. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  10. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  11. 2019Structured Pruning of Large Language Modelscited by 201
  12. 2017SMASH: One-Shot Model Architecture Search through HyperNetworkscited by 200
  13. 2020BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Modelscited by 187
  14. 2019The Evolved Transformercited by 180
  15. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145
  16. 2020HAT: Hardware-Aware Transformers for Efficient Natural Language Processingcited by 136
  17. 2017Genetic CNNcited by 124
  18. 2020AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Searchcited by 51
  19. 2020Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNetscited by 49
  20. 2019Small and Practical BERT Models for Sequence Labelingcited by 40
  21. 2020schuBERT: Optimizing Elements of BERTcited by 17
  22. 2020Improving Task-Agnostic BERT Distillation with Layer Mapping Searchcited by 8
  23. 2019Balanced One-shot Neural Architecture Optimizationcited by 6
  24. 2021Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillationcited by 6

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 11; 2023: 12; 2024: 5; 2025: 2.

31 papers.

Sort:

By area:

  1. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  2. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  3. 2023Speculative Decoding with Big Little Decodercited by 163
  4. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  5. 2023Neural Architecture Search: Insights from 1000 Paperscited by 136
  6. 2024Evolutionary Computation in the Era of Large Language Model: Survey and Roadmapcited by 89
  7. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  8. 2021Towards Efficient Post-training Quantization of Pre-trained Language Modelscited by 37
  9. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  10. 2023Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Reviewcited by 32
  11. 2023LLM Performance Predictors are good initializers for Architecture Searchcited by 29
  12. 2023Gradient-Free Structured Pruning with Unlabeled Datacited by 26
19 more
  1. 2024EvoPress: Accurate Dynamic Model Compression via Evolutionary Searchcited by 20
  2. 2022Extreme Compression for Pre-trained Transformers Made Simple and Efficientcited by 19
  3. 2022Exploring Extreme Parameter Compression for Pre-trained Language Modelscited by 15
  4. 2024A Comprehensive Survey of Compression Algorithms for Language Modelscited by 15
  5. 2022LiteTransformerSearch: Training-free Neural Architecture Search for Efficient Language Modelscited by 14
  6. 2022Differentially Private Model Compressioncited by 13
  7. 2022FlexiBERT: Are Current Transformer Architectures too Homogeneous and Rigid?cited by 12
  8. 2022AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translationcited by 9
  9. 2022Gradient-based Intra-attention Pruning on Pre-trained Language Modelscited by 9
  10. 2023Training-free Neural Architecture Search for RNNs and Transformerscited by 9
  11. 2023Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with Architecture-Routed Mixture-of-Expertscited by 7
  12. 2025Large Language Models and Evolutionary Computation: A Critical Review of Bidirectional Interaction, Automated Algorithm Design, and Co-Adaptive Systemscited by 6
  13. 2023MUX-PLMs: Data Multiplexing for High-throughput Language Modelscited by 4
  14. 2025HyperSHAP: Shapley Values and Interactions for Explaining Hyperparameter Optimizationcited by 4
  15. 2022AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Modelscited by 3
  16. 2024Merino: Entropy-driven Design for Generative Language Models on IoT Devicescited by 3
  17. 2023TransCODE: Co-design of Transformers and Accelerators for Efficient Training and Inferencecited by 1
  18. 2023YFlows: Systematic Dataflow Exploration and Code Generation for Efficient Neural Network Inference using SIMD Architectures on CPUscited by 1
  19. 2022SpeedLimit: Neural Architecture Search for Quantized Transformer Modelscited by 0