mlsanityData to 14 Oct 2026

To Asymmetry and Beyond: Structured Pruning of Sequence to Sequence Models for Improved Inference Efficiency

arXiv 2304.02721 ↗First posted 5 Apr 2023cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Sequence-to-sequence language models can be used to produce abstractive summaries which are coherent, relevant, and concise. Still, model sizes can make deployment in latency-sensitive or web-scale implementations difficult. This paper studies the relationship between model size, structured pruning, inference efficiency, and summarization accuracy on widely used summarization datasets. We show that model accuracy is tied to the encoder size while inference efficiency is connected to the decoder. Using asymmetric pruning can lead to nearly 3x improvement in inference latency with ~1 point loss in Rouge-2. Moreover, we find both the average degradation and the role of asymmetry to be consistent across model sizes and variations in datasets.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 39 of its 46 references that are in the map. First posted on arXiv 5 Apr 2023.

Builds on

39 of 46 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  11. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  12. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
27 more
  1. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  2. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  3. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  4. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  5. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  6. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  7. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  8. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  9. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 376
  10. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
  11. 2022Confident Adaptive Language Modelingcited by 310
  12. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  13. 2021PAQ: 65 Million Probably-Asked Questions and What You Can Do With Themcited by 200
  14. 2021Block Pruning For Faster Transformerscited by 177
  15. 2021Learned Token Pruning for Transformerscited by 139
  16. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  17. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 106
  18. 2020Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translationcited by 105
  19. 2020Pre-trained Summarization Distillationcited by 71
  20. 2020Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformerscited by 67
  21. 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
  22. 2019Scheduled Sampling for Transformerscited by 48
  23. 2022Scaling Laws for a Multi-Agent Reinforcement Learning Modelcited by 38
  24. 2020On the Predictability of Pruning Across Scalescited by 33
  25. 2022DQ-BART: Efficient Sequence-to-Sequence Model via Joint Distillation and Quantizationcited by 23
  26. 2022Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Modelscited by 20
  27. 2020A Streaming Approach For Efficient Batched Beam Searchcited by 1

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decodingcited by 40