# Dynamic-TinyBERT: Boost TinyBERT's Inference Efficiency by Dynamic Sequence Length

arXiv: https://arxiv.org/abs/2111.09645 · HTML: https://mlsanity.com/paper/2111.09645/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 14 references that are in the map.
- First posted on arXiv 18 Nov 2021.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [DynaBERT: Dynamic BERT with Adaptive Width and Depth](https://mlsanity.com/paper/2004.04037/) (2020)
- [Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing](https://mlsanity.com/paper/2006.03236/) (2020)
- [The Right Tool for the Job: Matching Model and Instance Complexities](https://mlsanity.com/paper/2004.07453/) (2020)
- [Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search](https://mlsanity.com/paper/2010.07003/) (2020)

## Built on by (most cited first, 4 of 4)

- [TangoBERT: Reducing Inference Cost by using Cascaded Architecture](https://mlsanity.com/paper/2204.06271/) (2022)
- [AttMEMO : Accelerating Transformers with Memoization on Big Memory Systems](https://mlsanity.com/paper/2301.09262/) (2023)
- [Inference Optimizations for Large Language Models: Effects, Challenges, and Practical Considerations](https://mlsanity.com/paper/2408.03130/) (2024)
- [QuaLA-MiniLM: a Quantized Length Adaptive MiniLM](https://mlsanity.com/paper/2210.17114/) (2022)

## Abstract (arXiv)

Limited computational budgets often prevent transformers from being used in production and from having their high accuracy utilized. TinyBERT addresses the computational efficiency by self-distilling BERT into a smaller transformer representation having fewer layers and smaller internal embedding. However, TinyBERT's performance drops when we reduce the number of layers by 50%, and drops even more abruptly when we reduce the number of layers by 75% for advanced NLP tasks such as span question answering. Additionally, a separate model must be trained for each inference scenario with its distinct computational budget. In this work we present Dynamic-TinyBERT, a TinyBERT model that utilizes sequence-length reduction and Hyperparameter Optimization for enhanced inference efficiency per any computational budget. Dynamic-TinyBERT is trained only once, performing on-par with BERT and achieving an accuracy-speedup trade-off superior to any other efficient approaches (up to 3.3x with <1% loss-drop). Upon publication, the code to reproduce our work will be open-sourced.