# NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM

arXiv: https://arxiv.org/abs/2110.15766 · HTML: https://mlsanity.com/paper/2110.15766/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 54 references that are in the map.
- First posted on arXiv 28 Oct 2021.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Learning Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1608.03665/) (2016)
- [Patient Knowledge Distillation for BERT Model Compression](https://mlsanity.com/paper/1908.09355/) (2019)
- [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://mlsanity.com/paper/2004.02984/) (2020)
- [Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT](https://mlsanity.com/paper/1909.05840/) (2019)
- [Measuring Bias in Contextualized Word Representations](https://mlsanity.com/paper/1906.07337/) (2019)
- [The Lottery Ticket Hypothesis for Pre-trained BERT Networks](https://mlsanity.com/paper/2007.12223/) (2020)
- [Accelerating Sparse Deep Neural Networks](https://mlsanity.com/paper/2104.08378/) (2021)
- [Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning](https://mlsanity.com/paper/2002.08307/) (2020)
- [Social Biases in NLP Models as Barriers for Persons with Disabilities](https://mlsanity.com/paper/2005.00813/) (2020)
- [Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch](https://mlsanity.com/paper/2102.04010/) (2021)
- [BinaryBERT: Pushing the Limit of BERT Quantization](https://mlsanity.com/paper/2012.15701/) (2020)
- [Characterising Bias in Compressed Models](https://mlsanity.com/paper/2010.03058/) (2020)
- [TVM: An Automated End-to-End Optimizing Compiler for Deep Learning](https://mlsanity.com/paper/1802.04799/) (2018)
- [PCONV: The Missing but Desirable Sparsity in DNN Weight Pruning for Real-time Execution on Mobile Devices](https://mlsanity.com/paper/1909.05073/) (2019)
- [LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression](https://mlsanity.com/paper/2004.04124/) (2020)
- [ADMM-NN: An Algorithm-Hardware Co-Design Framework of DNNs Using Alternating Direction Method of Multipliers](https://mlsanity.com/paper/1812.11677/) (2018)
- [Reweighted Proximal Pruning for Large-Scale Language Representation](https://mlsanity.com/paper/1909.12486/) (2019)
- [Neural Network Distiller: A Python Package For DNN Compression Research](https://mlsanity.com/paper/1910.12232/) (2019)
- [Progressive Weight Pruning of Deep Neural Networks using ADMM](https://mlsanity.com/paper/1810.07378/) (2018)

## Built on by (most cited first, 19 of 19)

- [Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2303.15647/) (2023)
- [Low-Rank Few-Shot Adaptation of Vision-Language Models](https://mlsanity.com/paper/2405.18541/) (2024)
- [DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution](https://mlsanity.com/paper/2405.17357/) (2024)
- [A Survey of Techniques for Optimizing Transformer Inference](https://mlsanity.com/paper/2307.07982/) (2023)
- [STEP: Learning N:M Structured Sparsity Masks from Scratch with Precondition](https://mlsanity.com/paper/2302.01172/) (2023)
- [Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey](https://mlsanity.com/paper/2205.08099/) (2022)
- [Minimum Variance Unbiased N:M Sparsity for the Neural Gradients](https://mlsanity.com/paper/2203.10991/) (2022)
- [DSFormer: Effective Compression of Text-Transformers by Dense-Sparse Weight Factorization](https://mlsanity.com/paper/2312.13211/) (2023)
- [CLIP-SVD: Efficient and Interpretable Vision-Language Adaptation via Singular Values](https://mlsanity.com/paper/2509.03740/) (2025)
- [Compressing Pre-trained Transformers via Low-Bit NxM Sparsity for Natural Language Understanding](https://mlsanity.com/paper/2206.15014/) (2022)
- [A Convex-optimization-based Layer-wise Post-training Pruner for Large Language Models](https://mlsanity.com/paper/2408.03728/) (2024)
- [Samoyeds: Accelerating MoE Models with Structured Sparsity Leveraging Sparse Tensor Cores](https://mlsanity.com/paper/2503.10725/) (2025)
- [Improvise, Adapt, Overcome -- Telescopic Adapters for Efficient Fine-tuning of Vision Language Models in Medical Imaging](https://mlsanity.com/paper/2512.13855/) (2025)
- [Privacy Amplification by Iteration for ADMM with (Strongly) Convex Objective Functions](https://mlsanity.com/paper/2312.08685/) (2023)
- [An Efficient Sparse Fine-Tuning with Low Quantization Error via Neural Network Pruning](https://mlsanity.com/paper/2502.11439/) (2025)
- [SPAP: Structured Pruning via Alternating Optimization and Penalty Methods](https://mlsanity.com/paper/2505.03373/) (2025)
- [MaskPro: Linear-Space Probabilistic Learning for Strict (N:M)-Sparsity on LLMs](https://mlsanity.com/paper/2506.12876/) (2025)
- [NanoQuant: Efficient Sub-1-Bit Quantization of Large Language Models](https://mlsanity.com/paper/2602.06694/) (2026)
- [Archtree: on-the-fly tree-structured exploration for latency-aware pruning of deep neural networks](https://mlsanity.com/paper/2311.10549/) (2023)

## Abstract (arXiv)

Natural Language Processing (NLP) has recently achieved success by using huge pre-trained Transformer networks. However, these models often contain hundreds of millions or even billions of parameters, bringing challenges to online deployment due to latency constraints. Recently, hardware manufacturers have introduced dedicated hardware for NxM sparsity to provide the flexibility of unstructured pruning with the runtime efficiency of structured approaches. NxM sparsity permits arbitrarily selecting M parameters to retain from a contiguous group of N in the dense representation. However, due to the extremely high complexity of pre-trained models, the standard sparse fine-tuning techniques often fail to generalize well on downstream tasks, which have limited data resources. To address such an issue in a principled manner, we introduce a new learning framework, called NxMTransformer, to induce NxM semi-structured sparsity on pretrained language models for natural language understanding to obtain better performance. In particular, we propose to formulate the NxM sparsity as a constrained optimization problem and use Alternating Direction Method of Multipliers (ADMM) to optimize the downstream tasks while taking the underlying hardware constraints into consideration. ADMM decomposes the NxM sparsification problem into two sub-problems that can be solved sequentially, generating sparsified Transformer networks that achieve high accuracy while being able to effectively execute on newly released hardware. We apply our approach to a wide range of NLP tasks, and our proposed method is able to achieve 1.7 points higher accuracy in GLUE score than current practices. Moreover, we perform detailed analysis on our approach and shed light on how ADMM affects fine-tuning accuracy for downstream tasks. Finally, we illustrate how NxMTransformer achieves performance improvement with knowledge distillation.