# PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination

arXiv: https://arxiv.org/abs/2001.08950 · HTML: https://mlsanity.com/paper/2001.08950/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jan 2020.

## Built on by (most cited first, 10 of 10)

- [A Survey of Resource-efficient LLM and Multimodal Foundation Models](https://mlsanity.com/paper/2401.08092/) (2024)
- [TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference](https://mlsanity.com/paper/2105.11618/) (2021)
- [Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications](https://mlsanity.com/paper/2204.11786/) (2022)
- [X-Former: In-Memory Acceleration of Transformers](https://mlsanity.com/paper/2303.07470/) (2023)
- [Block-Skim: Efficient Question Answering for Transformer](https://mlsanity.com/paper/2112.08560/) (2021)
- [The Need for Speed: Pruning Transformers with One Recipe](https://mlsanity.com/paper/2403.17921/) (2024)
- [KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head](https://mlsanity.com/paper/2410.00161/) (2024)
- [Efficient Vision Transformer for Human Pose Estimation via Patch Selection](https://mlsanity.com/paper/2306.04225/) (2023)
- [CATP: Cross-Attention Token Pruning for Accuracy Preserved Multimodal Model Inference](https://mlsanity.com/paper/2404.08567/) (2024)
- [Transformers with Joint Tokens and Local-Global Attention for Efficient Human Pose Estimation](https://mlsanity.com/paper/2503.00232/) (2025)

## Abstract (arXiv)

We develop a novel method, called PoWER-BERT, for improving the inference time of the popular BERT model, while maintaining the accuracy. It works by: a) exploiting redundancy pertaining to word-vectors (intermediate encoder outputs) and eliminating the redundant vectors. b) determining which word-vectors to eliminate by developing a strategy for measuring their significance, based on the self-attention mechanism. c) learning how many word-vectors to eliminate by augmenting the BERT model and the loss function. Experiments on the standard GLUE benchmark shows that PoWER-BERT achieves up to 4.5x reduction in inference time over BERT with <1% loss in accuracy. We show that PoWER-BERT offers significantly better trade-off between accuracy and inference time compared to prior methods. We demonstrate that our method attains up to 6.8x reduction in inference time with <1% loss in accuracy when applied over ALBERT, a highly compressed version of BERT. The code for PoWER-BERT is publicly available at https://github.com/IBM/PoWER-BERT.