mlsanityData to 14 Oct 2026

NASH: A Simple Unified Framework of Structured Pruning for Accelerating Encoder-Decoder Language Models

arXiv 2310.10054 ↗First posted 16 Oct 2023cs.CL cs.AI cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Structured pruning methods have proven effective in reducing the model size and accelerating inference speed in various network architectures such as Transformers. Despite the versatility of encoder-decoder models in numerous NLP tasks, the structured pruning methods on such models are relatively less explored compared to encoder-only models. In this study, we investigate the behavior of the structured pruning of the encoder-decoder models in the decoupled pruning perspective of the encoder and decoder component, respectively. Our findings highlight two insights: (1) the number of decoder layers is the dominant factor of inference speed, and (2) low sparsity in the pruned encoder network enhances generation quality. Motivated by these findings, we propose a simple and effective framework, NASH, that narrows the encoder and shortens the decoder networks of encoder-decoder models. Extensive experiments on diverse generation and inference tasks validate the effectiveness of our method in both speedup and output quality.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 49 of its 60 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

49 of 60 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  11. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
37 more
  1. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  2. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  3. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  4. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  5. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  6. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  7. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  8. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  9. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  10. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  11. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  12. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 361
  13. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  14. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  15. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  16. 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 295
  17. 2022UL2: Unifying Language Learning Paradigmscited by 295
  18. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  19. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  20. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  21. 2020Revisiting Few-sample BERT Fine-tuningcited by 238
  22. 2019Structured Pruning of Large Language Modelscited by 203
  23. 2021Block Pruning For Faster Transformerscited by 177
  24. 2022Structured Pruning Learns Compact and Accurate Modelscited by 170
  25. 2022A Fast Post-Training Pruning Framework for Transformerscited by 162
  26. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 139
  27. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  28. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 106
  29. 2020Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translationcited by 105
  30. 2020Pre-trained Summarization Distillationcited by 71
  31. 2020Compression of Deep Learning Models for Text: A Surveycited by 64
  32. 2019TWEETQA: A Social Media Focused Question Answering Datasetcited by 55
  33. 2023What Matters In The Structured Pruning of Generative Language Models?cited by 30
  34. 2021EfficientBERT: Progressively Searching Multilayer Perceptron via Warm-up Knowledge Distillationcited by 15
  35. 2016Effective Quantization Methods for Recurrent Neural Networkscited by 14
  36. 2018Effective Quantization Approaches for Recurrent Neural Networkscited by 10
  37. 2022TextPruner: A Model Pruning Toolkit for Pre-Trained Language Modelscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 3; 2026: 1.

7 papers.

  1. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 84
  2. 2024Designing Large Foundation Models for Efficient Training and Inference: A Surveycited by 7
  3. 2024Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradientcited by 6
  4. 2024HESSO: Towards Automatic Efficient and User Friendly Any Neural Network Training and Pruningcited by 4
  5. 2025PASER: Post-Training Data Selection for Efficient Pruned Large Language Model Recoverycited by 4
  6. 2026GradPruner: Gradient-Guided Layer Pruning Enabling Efficient Fine-Tuning and Inference for LLMscited by 3
  7. 2025MaskPro: Linear-Space Probabilistic Learning for Strict (N:M)-Sparsity on LLMscited by 1