# Optimal Subarchitecture Extraction For BERT

arXiv: https://arxiv.org/abs/2010.10499 · HTML: https://mlsanity.com/paper/2010.10499/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Oct 2020.

## Built on by (most cited first, 9 of 9)

- [The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models](https://mlsanity.com/paper/2203.07259/) (2022)
- [How to Train BERT with an Academic Budget](https://mlsanity.com/paper/2104.07705/) (2021)
- [NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework](https://mlsanity.com/paper/2111.04130/) (2021)
- [Curriculum learning for language modeling](https://mlsanity.com/paper/2108.02170/) (2021)
- [Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation](https://mlsanity.com/paper/2104.11928/) (2021)
- [RobBERTje: a Distilled Dutch BERT Model](https://mlsanity.com/paper/2204.13511/) (2022)
- [A Comprehensive Comparison of Pre-training Language Models](https://mlsanity.com/paper/2106.11483/) (2021)
- [Compressing Cross-Lingual Multi-Task Models at Qualtrics](https://mlsanity.com/paper/2211.15927/) (2022)
- [Unbiased Sentence Encoder For Large-Scale Multi-lingual Search Engines](https://mlsanity.com/paper/2106.07719/) (2021)

## Abstract (arXiv)

We extract an optimal subset of architectural parameters for the BERT architecture from Devlin et al. (2018) by applying recent breakthroughs in algorithms for neural architecture search. This optimal subset, which we refer to as "Bort", is demonstrably smaller, having an effective (that is, not counting the embedding layer) size of 5.5% the original BERT-large architecture, and 16% of the net size. Bort is also able to be pretrained in 288 GPU hours, which is 1.2% of the time required to pretrain the highest-performing BERT parametric architectural variant, RoBERTa-large (Liu et al., 2019), and about 33% of that of the world-record, in GPU hours, required to train BERT-large on the same hardware. It is also 7.9x faster on a CPU, as well as being better performing than other compressed variants of the architecture, and some of the non-compressed variants: it obtains performance improvements of between 0.3% and 31%, absolute, with respect to BERT-large, on multiple public natural language understanding (NLU) benchmarks.