mlsanityData to 7 Oct 2026

Attention Temperature Matters in Abstractive Summarization Distillation

arXiv 2106.03441 ↗First posted 7 Jun 2021cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

Recent progress of abstractive text summarization largely relies on large pre-trained sequence-to-sequence Transformer models, which are computationally expensive. This paper aims to distill these large models into smaller ones for faster inference and minimal performance loss. Pseudo-labeling based methods are popular in sequence-to-sequence model distillation. In this paper, we find simply manipulating attention temperatures in Transformers can make pseudo labels easier to learn for student models. Our experiments on three summarization datasets show our proposed method consistently improves over vanilla pseudo-labeling based methods. We also find that both the pseudo labels and summaries produced by our students are shorter and more abstractive. Our code is available at urlhttps://github.com/Shengqiang-Zhang/plate.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 38 of its 51 references that are in the map. First posted on arXiv 7 Jun 2021.

Builds on

38 of 51 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  12. 2014FitNets: Hints for Thin Deep Netscited by 1,619
26 more
  1. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  2. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  3. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  4. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  5. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  6. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  7. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  8. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  9. 2015Teaching Machines to Read and Comprehendcited by 1,039
  10. 2016Sequence-Level Knowledge Distillationcited by 761
  11. 2019Text Summarization with Pretrained Encoderscited by 553
  12. 2018Born Again Neural Networkscited by 539
  13. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  14. 2013Do Deep Nets Really Need to be Deep?cited by 466
  15. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  16. 2017Non-Autoregressive Neural Machine Translationcited by 457
  17. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  18. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  19. 2019Revisiting Self-Training for Neural Sequence Generationcited by 154
  20. 2017Sobolev Training for Neural Networkscited by 136
  21. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 112
  22. 2020Pre-trained Summarization Distillationcited by 70
  23. 2016Learning-Based Single-Document Summarization with Compression and Anaphoricity Constraintscited by 34
  24. 2020Understanding Neural Abstractive Summarization Models via Uncertaintycited by 34
  25. 2020Noisy Self-Knowledge Distillation for Text Summarizationcited by 31
  26. 2017Stronger Baselines for Trustable Results in Neural Machine Translationcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 4; 2025: 3; 2026: 2.

14 papers.

Sort:

By area:

  1. 2024Learning to Summarize from LLM-generated Feedbackcited by 24
  2. 2023A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Trainingcited by 16
  3. 2023Should We Attend More or Less? Modulating Attention for Fairnesscited by 11
  4. 2025Mitigating Hallucinations in Diffusion Models through Adaptive Attention Modulationcited by 11
  5. 2024TriSum: Learning Summarization Ability from Large Language Models with Structured Rationalecited by 10
  6. 2023Enhancing Abstractiveness of Summarization Models through Calibrated Distillationcited by 8
  7. 2024A Bayesian Flow Network Framework for Chemistry Taskscited by 6
  8. 2026Attention Smoothing Is All You Need For Unlearningcited by 6
  9. 2023Task-agnostic Distillation of Encoder-Decoder Language Modelscited by 4
  10. 2023Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inferencecited by 4
  11. 2024Understanding Position Bias Effects on Fairness in Social Multi-Document Summarizationcited by 4
  12. 2025Optimal Attention Temperature Improves the Robustness of In-Context Learning under Distribution Shift in High Dimensionscited by 1
2 more
  1. 2025InforME: Improving Informativeness of Abstractive Text Summarization With Informative Attention Guided by Named Entity Saliencecited by 0
  2. 2026CoANeRV: Coordinate-Aware Token-Space Neural Video Representationcited by 0