# AUBER: Automated BERT Regularization

arXiv: https://arxiv.org/abs/2009.14409 · HTML: https://mlsanity.com/paper/2009.14409/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Sep 2020.

## Built on by (most cited first, 4 of 4)

- [A Comprehensive Survey of Compression Algorithms for Language Models](https://mlsanity.com/paper/2401.15347/) (2024)
- [Unifying Uniform and Binary-coding Quantization for Accurate Compression of Large Language Models](https://mlsanity.com/paper/2506.03781/) (2025)
- [Accurate Sublayer Pruning for Large Language Models by Exploiting Latency and Tunability Information](https://mlsanity.com/paper/2506.03510/) (2025)
- [Can persistent homology whiten Transformer-based black-box models? A case study on BERT compression](https://mlsanity.com/paper/2312.10702/) (2023)

## Abstract (arXiv)

How can we effectively regularize BERT? Although BERT proves its effectiveness in various downstream natural language processing tasks, it often overfits when there are only a small number of training instances. A promising direction to regularize BERT is based on pruning its attention heads based on a proxy score for head importance. However, heuristic-based methods are usually suboptimal since they predetermine the order by which attention heads are pruned. In order to overcome such a limitation, we propose AUBER, an effective regularization method that leverages reinforcement learning to automatically prune attention heads from BERT. Instead of depending on heuristics or rule-based policies, AUBER learns a pruning policy that determines which attention heads should or should not be pruned for regularization. Experimental results show that AUBER outperforms existing pruning methods by achieving up to 10% better accuracy. In addition, our ablation study empirically demonstrates the effectiveness of our design choices for AUBER.