# schuBERT: Optimizing Elements of BERT

arXiv: https://arxiv.org/abs/2005.06628 · HTML: https://mlsanity.com/paper/2005.06628/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 May 2020.

## Built on by (most cited first, 17 of 17)

- [A Fast Post-Training Pruning Framework for Transformers](https://mlsanity.com/paper/2204.09656/) (2022)
- [Learned Token Pruning for Transformers](https://mlsanity.com/paper/2107.00910/) (2021)
- [Model Compression and Efficient Inference for Large Language Models: A Survey](https://mlsanity.com/paper/2402.09748/) (2024)
- [AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models](https://mlsanity.com/paper/2107.13686/) (2021)
- [Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators](https://mlsanity.com/paper/2106.02205/) (2021)
- [Gradient-Free Structured Pruning with Unlabeled Data](https://mlsanity.com/paper/2303.04185/) (2023)
- [From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compression](https://mlsanity.com/paper/2112.07198/) (2021)
- [ProtAugment: Unsupervised diverse short-texts paraphrasing for intent detection meta-learning](https://mlsanity.com/paper/2105.12995/) (2021)
- [FlexiBERT: Are Current Transformer Architectures too Homogeneous and Rigid?](https://mlsanity.com/paper/2205.11656/) (2022)
- [StorSeismic: A new paradigm in deep learning for seismic processing](https://mlsanity.com/paper/2205.00222/) (2022)
- [EdgeTran: Co-designing Transformers for Efficient Inference on Mobile Edge Platforms](https://mlsanity.com/paper/2303.13745/) (2023)
- [Bilaterally Slimmable Transformer for Elastic and Efficient Visual Question Answering](https://mlsanity.com/paper/2203.12814/) (2022)
- [TransCODE: Co-design of Transformers and Accelerators for Efficient Training and Inference](https://mlsanity.com/paper/2303.14882/) (2023)
- [Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attention](https://mlsanity.com/paper/2310.07911/) (2023)
- [Learning Dynamic BERT via Trainable Gate Variables and a Bi-modal Regularizer](https://mlsanity.com/paper/2102.09727/) (2021)
- [LV-BERT: Exploiting Layer Variety for BERT](https://mlsanity.com/paper/2106.11740/) (2021)
- [The silence of the weights: a structural pruning strategy for attention-based audio signal architectures with second order metrics](https://mlsanity.com/paper/2509.26207/) (2025)

## Abstract (arXiv)

Transformers citepvaswani2017attention have gradually become a key component for many state-of-the-art natural language representation models. A recent Transformer based model- BERT citepdevlin2018bert achieved state-of-the-art results on various natural language processing tasks, including GLUE, SQuAD v1.1, and SQuAD v2.0. This model however is computationally prohibitive and has a huge number of parameters. In this work we revisit the architecture choices of BERT in efforts to obtain a lighter model. We focus on reducing the number of parameters yet our methods can be applied towards other objectives such FLOPs or latency. We show that much efficient light BERT models can be obtained by reducing algorithmically chosen correct architecture design dimensions rather than reducing the number of Transformer encoder layers. In particular, our schuBERT gives 6.6% higher average accuracy on GLUE and SQuAD datasets as compared to BERT with three encoder layers while having the same number of parameters.