# MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models

arXiv: https://arxiv.org/abs/2409.17481 · HTML: https://mlsanity.com/paper/2409.17481/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 47 references that are in the map.
- First posted on arXiv 26 Sep 2024.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [Importance Estimation for Neural Network Pruning](https://mlsanity.com/paper/1906.10771/) (2019)
- [Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning](https://mlsanity.com/paper/2310.06694/) (2023)
- [SliceGPT: Compress Large Language Models by Deleting Rows and Columns](https://mlsanity.com/paper/2401.15024/) (2024)
- [ShortGPT: Layers in Large Language Models are More Redundant Than You Expect](https://mlsanity.com/paper/2403.03853/) (2024)
- [Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time](https://mlsanity.com/paper/2310.17157/) (2023)
- [Accelerating Sparse Deep Neural Networks](https://mlsanity.com/paper/2104.08378/) (2021)
- [Structured Pruning of Large Language Models](https://mlsanity.com/paper/1910.04732/) (2019)
- [Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch](https://mlsanity.com/paper/2102.04010/) (2021)
- [Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity](https://mlsanity.com/paper/2310.05175/) (2023)
- [The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter](https://mlsanity.com/paper/2306.03805/) (2023)

## Built on by (most cited first, 30 of 50)

- [TinyFusion: Diffusion Transformers Learned Shallow](https://mlsanity.com/paper/2412.01199/) (2024)
- [A Survey on Efficient Vision-Language-Action Models](https://mlsanity.com/paper/2510.24795/) (2025)
- [SepPrune: Structured Pruning for Efficient Deep Speech Separation](https://mlsanity.com/paper/2505.12079/) (2025)
- [QVGen: Pushing the Limit of Quantized Video Generative Models](https://mlsanity.com/paper/2505.11497/) (2025)
- [On-Device Diffusion Transformer Policy for Efficient Robot Manipulation](https://mlsanity.com/paper/2508.00697/) (2025)
- [SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression](https://mlsanity.com/paper/2410.09615/) (2024)
- [2SSP: A Two-Stage Framework for Structured Pruning of LLMs](https://mlsanity.com/paper/2501.17771/) (2025)
- [ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs](https://mlsanity.com/paper/2502.00258/) (2025)
- [Learnable Sparsity for Vision Generative Models](https://mlsanity.com/paper/2412.02852/) (2024)
- [DenoiseRotator: Enhance Pruning Robustness for LLMs via Importance Concentration](https://mlsanity.com/paper/2505.23049/) (2025)
- [La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation](https://mlsanity.com/paper/2507.01299/) (2025)
- [NIRVANA: Structured Pruning Reimagined for Large Language Model Compression](https://mlsanity.com/paper/2509.14230/) (2025)
- [GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling](https://mlsanity.com/paper/2604.18556/) (2026)
- [The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMM](https://mlsanity.com/paper/2510.01650/) (2025)
- [A Proximal Operator for Inducing 2:4-Sparsity](https://mlsanity.com/paper/2501.18015/) (2025)
- [Pivoting Factorization: A Compact Meta Low-Rank Representation of Sparsity for Efficient Inference in Large Language Models](https://mlsanity.com/paper/2501.19090/) (2025)
- [PATCH: Learnable Tile-level Hybrid Sparsity for LLMs](https://mlsanity.com/paper/2509.23410/) (2025)
- [Expand Neurons, Not Parameters](https://mlsanity.com/paper/2510.04500/) (2025)
- [ARMOR: High-Performance Semi-Structured Pruning via Adaptive Matrix Factorization](https://mlsanity.com/paper/2510.05528/) (2025)
- [MC#: Mixture Compressor for Mixture-of-Experts Large Models](https://mlsanity.com/paper/2510.10962/) (2025)
- [OPTIMA: Optimal One-shot Pruning for LLMs via Quadratic Programming Reconstruction](https://mlsanity.com/paper/2512.13886/) (2025)
- [Train Less, Infer Faster: Efficient Model Finetuning and Compression via Structured Sparsity](https://mlsanity.com/paper/2602.09169/) (2026)
- [Deterministic Differentiable Structured Pruning for Large Language Models](https://mlsanity.com/paper/2603.08065/) (2026)
- [Mechanism-Driven Monitors for Preemptive Detection of LLM Training Instability](https://mlsanity.com/paper/2606.28116/) (2026)
- [CoNNect: Connectivity-Based Regularization for Structural Pruning](https://mlsanity.com/paper/2502.00744/) (2025)
- [SPAP: Structured Pruning via Alternating Optimization and Penalty Methods](https://mlsanity.com/paper/2505.03373/) (2025)
- [MaskPro: Linear-Space Probabilistic Learning for Strict (N:M)-Sparsity on LLMs](https://mlsanity.com/paper/2506.12876/) (2025)
- [CAST: Continuous and Differentiable Semi-Structured Sparsity-Aware Training for Large Language Models](https://mlsanity.com/paper/2509.25996/) (2025)
- [PermLLM: Learnable Channel Permutation for N:M Sparse Large Language Models](https://mlsanity.com/paper/2510.10136/) (2025)
- [Sparse by Rule: Probability-Based N:M Pruning for Spiking Neural Networks](https://mlsanity.com/paper/2511.12097/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) are distinguished by their massive parameter counts, which typically result in significant redundancy. This work introduces MaskLLM, a learnable pruning method that establishes Semi-structured (or ``N:M'') Sparsity in LLMs, aimed at reducing computational overhead during inference. Instead of developing a new importance criterion, MaskLLM explicitly models N:M patterns as a learnable distribution through Gumbel Softmax sampling. This approach facilitates end-to-end training on large-scale datasets and offers two notable advantages: 1) High-quality Masks - our method effectively scales to large datasets and learns accurate masks; 2) Transferability - the probabilistic modeling of mask distribution enables the transfer learning of sparsity across domains or tasks. We assessed MaskLLM using 2:4 sparsity on various LLMs, including LLaMA-2, Nemotron-4, and GPT-3, with sizes ranging from 843M to 15B parameters, and our empirical results show substantial improvements over state-of-the-art methods. For instance, leading approaches achieve a perplexity (PPL) of 10 or greater on Wikitext compared to the dense model's 5.12 PPL, but MaskLLM achieves a significantly lower 6.72 PPL solely by learning the masks with frozen weights. Furthermore, MaskLLM's learnable nature allows customized masks for lossless application of 2:4 sparsity to downstream tasks or domains. Code is available at https://github.com/NVlabs/MaskLLM.