# Structured Pruning of a BERT-based Question Answering Model

arXiv: https://arxiv.org/abs/1910.06360 · HTML: https://mlsanity.com/paper/1910.06360/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Oct 2019.

## Built on by (most cited first, 16 of 16)

- [NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search](https://mlsanity.com/paper/2105.14444/) (2021)
- [AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning](https://mlsanity.com/paper/2102.01386/) (2021)
- [Layer-wise Model Pruning based on Mutual Information](https://mlsanity.com/paper/2108.12594/) (2021)
- [Causal Distillation for Language Models](https://mlsanity.com/paper/2112.02505/) (2021)
- [OSSCAR: One-Shot Structured Pruning in Vision and Language Models with Combinatorial Optimization](https://mlsanity.com/paper/2403.12983/) (2024)
- [KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization](https://mlsanity.com/paper/2101.05938/) (2021)
- [You Only Compress Once: Towards Effective and Elastic BERT Compression via Exploit-Explore Stochastic Nature Gradient](https://mlsanity.com/paper/2106.02435/) (2021)
- [Pyramid-BERT: Reducing Complexity via Successive Core-set based Token Selection](https://mlsanity.com/paper/2203.14380/) (2022)
- [MKQ-BERT: Quantized BERT with 4-bits Weights and Activations](https://mlsanity.com/paper/2203.13483/) (2022)
- [TextPruner: A Model Pruning Toolkit for Pre-Trained Language Models](https://mlsanity.com/paper/2203.15996/) (2022)
- [Gradient-based Intra-attention Pruning on Pre-trained Language Models](https://mlsanity.com/paper/2212.07634/) (2022)
- [DoT: An efficient Double Transformer for NLP tasks with tables](https://mlsanity.com/paper/2106.00479/) (2021)
- [Interactive Question Answering Systems: Literature Review](https://mlsanity.com/paper/2209.01621/) (2022)
- [On the Prunability of Attention Heads in Multilingual BERT](https://mlsanity.com/paper/2109.12683/) (2021)
- [Delta Keyword Transformer: Bringing Transformers to the Edge through Dynamically Pruned Multi-Head Self-Attention](https://mlsanity.com/paper/2204.03479/) (2022)
- [What to Do When Your Discrete Optimization Is the Size of a Neural Network?](https://mlsanity.com/paper/2402.10339/) (2024)

## Abstract (arXiv)

The recent trend in industry-setting Natural Language Processing (NLP) research has been to operate large %scale pretrained language models like BERT under strict computational limits. While most model compression work has focused on "distilling" a general-purpose language representation using expensive pretraining distillation, less attention has been paid to creating smaller task-specific language representations which, arguably, are more useful in an industry setting. In this paper, we investigate compressing BERT- and RoBERTa-based question answering systems by structured pruning of parameters from the underlying transformer model. We find that an inexpensive combination of task-specific structured pruning and task-specific distillation, without the expense of pretraining distillation, yields highly-performing models across a range of speed/accuracy tradeoff operating points. We start from existing full-size models trained for SQuAD 2.0 or Natural Questions and introduce gates that allow selected parts of transformers to be individually eliminated. Specifically, we investigate (1) structured pruning to reduce the number of parameters in each transformer layer, (2) applicability to both BERT- and RoBERTa-based models, (3) applicability to both SQuAD 2.0 and Natural Questions, and (4) combining structured pruning with distillation. We achieve a near-doubling of inference speed with less than a 0.5 F1-point loss in short answer accuracy on Natural Questions.