# Reweighted Proximal Pruning for Large-Scale Language Representation

arXiv: https://arxiv.org/abs/1909.12486 · HTML: https://mlsanity.com/paper/1909.12486/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Sep 2019.

## Built on by (most cited first, 29 of 29)

- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [Chasing Sparsity in Vision Transformers: An End-to-End Exploration](https://mlsanity.com/paper/2106.04533/) (2021)
- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [Unified Visual Transformer Compression](https://mlsanity.com/paper/2203.08243/) (2022)
- [Only Train Once: A One-Shot Neural Network Training And Pruning Framework](https://mlsanity.com/paper/2107.07467/) (2021)
- [Model Compression and Efficient Inference for Large Language Models: A Survey](https://mlsanity.com/paper/2402.09748/) (2024)
- [Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers](https://mlsanity.com/paper/2303.01610/) (2023)
- [Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigm](https://mlsanity.com/paper/2104.08682/) (2021)
- [FLAT: An Optimized Dataflow for Mitigating Attention Bottlenecks](https://mlsanity.com/paper/2107.06419/) (2021)
- [Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm](https://mlsanity.com/paper/2110.08190/) (2021)
- [Layer-wise Model Pruning based on Mutual Information](https://mlsanity.com/paper/2108.12594/) (2021)
- [RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillation](https://mlsanity.com/paper/2109.10164/) (2021)
- [NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM](https://mlsanity.com/paper/2110.15766/) (2021)
- [From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compression](https://mlsanity.com/paper/2112.07198/) (2021)
- [Structured Sparsity Inducing Adaptive Optimizers for Deep Learning](https://mlsanity.com/paper/2102.03869/) (2021)
- [Differentially Private Model Compression](https://mlsanity.com/paper/2206.01838/) (2022)
- [An Automatic and Efficient BERT Pruning for Edge AI Systems](https://mlsanity.com/paper/2206.10461/) (2022)
- [Efficient Token Compression for Vision Transformer with Spatial Information Preserved](https://mlsanity.com/paper/2503.23455/) (2025)
- [Breaking through Deterministic Barriers: Randomized Pruning Mask Generation and Selection](https://mlsanity.com/paper/2310.13183/) (2023)
- [DSFormer: Effective Compression of Text-Transformers by Dense-Sparse Weight Factorization](https://mlsanity.com/paper/2312.13211/) (2023)
- [Compressing Pre-trained Transformers via Low-Bit NxM Sparsity for Natural Language Understanding](https://mlsanity.com/paper/2206.15014/) (2022)
- [Efficient model compression with Random Operation Access Specific Tile (ROAST) hashing](https://mlsanity.com/paper/2207.10702/) (2022)
- [Demystifying Map Space Exploration for NPUs](https://mlsanity.com/paper/2210.03731/) (2022)
- [On the Compression of Natural Language Models](https://mlsanity.com/paper/2112.11480/) (2021)
- [Latency Adjustable Transformer Encoder for Language Understanding](https://mlsanity.com/paper/2201.03327/) (2022)
- [Can persistent homology whiten Transformer-based black-box models? A case study on BERT compression](https://mlsanity.com/paper/2312.10702/) (2023)
- [Structured Pattern Pruning Using Regularization](https://mlsanity.com/paper/2109.08814/) (2021)
- [SparseOptimizer: Sparsify Language Models through Moreau-Yosida Regularization and Accelerate via Compiler Co-design](https://mlsanity.com/paper/2306.15656/) (2023)
- [Downsized and Compromised?: Assessing the Faithfulness of Model Compression](https://mlsanity.com/paper/2510.06125/) (2025)

## Abstract (arXiv)

Recently, pre-trained language representation flourishes as the mainstay of the natural language understanding community, e.g., BERT. These pre-trained language representations can create state-of-the-art results on a wide range of downstream tasks. Along with continuous significant performance improvement, the size and complexity of these pre-trained neural models continue to increase rapidly. Is it possible to compress these large-scale language representation models? How will the pruned language representation affect the downstream multi-task transfer learning objectives? In this paper, we propose Reweighted Proximal Pruning (RPP), a new pruning method specifically designed for a large-scale language representation model. Through experiments on SQuAD and the GLUE benchmark suite, we show that proximal pruned BERT keeps high accuracy for both the pre-training task and the downstream multiple fine-tuning tasks at high prune ratio. RPP provides a new perspective to help us analyze what large-scale language representation might learn. Additionally, RPP makes it possible to deploy a large state-of-the-art language representation model such as BERT on a series of distinct devices (e.g., online servers, mobile phones, and edge devices).