mlsanityData to 7 Oct 2026

Differentiable Subset Pruning of Transformer Heads

arXiv 2108.04657 ↗First posted 10 Aug 2021cs.CLcited by 46

Not on the map. Why?

Abstract (arXiv)

Multi-head attention, a collection of several attention mechanisms that independently attend to different parts of the input, is the key ingredient in the Transformer. Recent work has shown, however, that a large proportion of the heads in a Transformer's multi-head attention mechanism can be safely pruned away without significantly harming the performance of the model; such pruning leads to models that are noticeably smaller and faster in practice. Our work introduces a new head pruning technique that we term differentiable subset pruning. Intuitively, our method learns per-head importance variables and then enforces a user-specified hard constraint on the number of unpruned heads. The importance variables are learned via stochastic gradient descent. We conduct experiments on natural language inference and machine translation; we show that differentiable subset pruning performs comparably or better than previous works while offering precise control of the sparsity level.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Builds on 47 of its 64 references that are in the map. First posted on arXiv 10 Aug 2021.

Builds on

47 of 64 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  10. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  11. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  12. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
35 more
  1. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  2. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  3. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  4. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  5. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  6. 2019Are Sixteen Heads Really Better than One?cited by 803
  7. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  8. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  9. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  10. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  11. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
  12. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
  13. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
  14. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  15. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  16. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  17. 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 291
  18. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  19. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  20. 2019Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacementcited by 205
  21. 2019Assessing BERT's Syntactic Abilitiescited by 191
  22. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  23. 2017Data-Driven Sparse Structure Selection for Deep Neural Networkscited by 181
  24. 2019Facebook FAIR's WMT19 News Translation Task Submissioncited by 179
  25. 2014A* Samplingcited by 150
  26. 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
  27. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  28. 2018Neural Nearest Neighbors Networkscited by 98
  29. 2018Faster gaze prediction with dense networks and Fisher pruningcited by 92
  30. 2017REBAR: Low-variance, unbiased gradient estimates for discrete latent variable modelscited by 78
  31. 2019Reparameterizable Subset Sampling via Continuous Relaxationscited by 69
  32. 2019Head-Driven Phrase Structure Grammar Parsing on Penn Treebankcited by 43
  33. 2019Learning Sparse Networks Using Targeted Dropoutcited by 40
  34. 2020Successfully Applying the Stabilized Lottery Ticket Hypothesis to the Transformer Architecturecited by 23
  35. 2017Probabilistic Typology: Deep Generative Models of Vowel Inventoriescited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 7; 2023: 15; 2024: 12; 2025: 7; 2026: 4.

46 papers.

Sort:

By area:

  1. 2023Continual Pre-training of Language Modelscited by 194
  2. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  3. 2024Transformers are Multi-State RNNscited by 120
  4. 2023X-Pruner: eXplainable Pruning for Vision Transformerscited by 66
  5. 2022Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scalecited by 64
  6. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  7. 2023Discrete Prompt Compression with Reinforcement Learningcited by 42
  8. 2023Transformers in Speech Processing: A Surveycited by 38
  9. 2024Activation Scaling for Steering and Interpreting Language Modelscited by 33
  10. 2023Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Modelscited by 25
  11. 2021Pruning Self-attentions into Convolutional Layers in Single Pathcited by 24
  12. 2023Faster Causal Attention Over Large Sequences Through Sparse Flash Attentioncited by 24
34 more
  1. 2023Adapting a Language Model While Preserving its General Knowledgecited by 23
  2. 2024Controllable Context Sensitivity and the Knob Behind Itcited by 22
  3. 2022How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformerscited by 20
  4. 2022Mokey: Enabling Narrow Fixed-Point Inference for Out-of-the-Box Floating-Point Transformer Modelscited by 16
  5. 2025Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformerscited by 16
  6. 2023Interpreting and Exploiting Functional Specialization in Multi-Head Attention under Multi-task Learningcited by 9
  7. 2023Tailoring Instructions to Student's Learning Levels Boosts Knowledge Distillationcited by 8
  8. 2025Identifying and Evaluating Inactive Heads in Pretrained LLMscited by 8
  9. 2022Probing via Promptingcited by 7
  10. 2023Sub-network Discovery and Soft-masking for Continual Learning of Mixed Taskscited by 7
  11. 2025Joint Localization and Activation Editing for Low-Resource Fine-Tuningcited by 7
  12. 2023PreQuant: A Task-agnostic Quantization Approach for Pre-trained Language Modelscited by 5
  13. 2024Sparsity-Accelerated Training for Large Language Modelscited by 5
  14. 2022Auto-Compressing Subset Pruning for Semantic Image Segmentationcited by 4
  15. 2023Token Sparsification for Faster Medical Image Segmentationcited by 4
  16. 2023Just CHOP: Embarrassingly Simple LLM Compressioncited by 4
  17. 2023Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inferencecited by 4
  18. 2024From PEFT to DEFT: Parameter Efficient Finetuning for Reducing Activation Density in Transformerscited by 4
  19. 2024Automatic Channel Pruning for Multi-Head Attentioncited by 3
  20. 2022Delta Keyword Transformer: Bringing Transformers to the Edge through Dynamically Pruned Multi-Head Self-Attentioncited by 2
  21. 2025Efficient Token Compression for Vision Transformer with Spatial Information Preservedcited by 2
  22. 2023Finding the Pillars of Strength for Multi-Head Attentioncited by 1
  23. 2024Tiny Reinforcement Learning for Quadruped Locomotion using Decision Transformerscited by 1
  24. 2024NeuroPrune: A Neuro-inspired Topological Sparse Training Algorithm for Large Language Modelscited by 1
  25. 2025SHRP: Specialized Head Routing and Pruning for Efficient Encoder Compressioncited by 1
  26. 2026Greedy-Gnorm: A Gradient Matrix Norm-Based Alternative to Attention Entropy for Head Pruningcited by 1
  27. 2026Accelerating Vision Foundation Models with Drop-in Depthwise Convolutioncited by 1
  28. 2024Speeding Up Image Classifiers with Little Companionscited by 0
  29. 2024(PASS) Visual Prompt Locates Good Structure Sparsity through a Recurrent HyperNetworkcited by 0
  30. 2024Identifying Sub-networks in Neural Networks via Functionally Similar Representationscited by 0
  31. 2025Understanding Differential Transformer Unchains Pretrained Self-Attentionscited by 0
  32. 2025Assortment of Attention Heads: Accelerating Federated PEFT with Head Pruning and Strategic Client Selectioncited by 0
  33. 2026Complementary Attention Head Pruning for Efficient Transformerscited by 0
  34. 2026Interpretability-Guided Soft Pruning of Attention Heads in Vision Transformerscited by 0