mlsanityData to 7 Oct 2026

EL-Attention: Memory Efficient Lossless Attention for Generation

arXiv 2105.04779 ↗First posted 11 May 2021cs.CL cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Transformer model with multi-head attention requires caching intermediate results for efficient inference in generation tasks. However, cache brings new memory-related costs and prevents leveraging larger batch size for faster speed. We propose memory-efficient lossless attention (called EL-attention) to address this issue. It avoids heavy operations for building multi-head keys and values, cache for them is not needed. EL-attention constructs an ensemble of attention results by expanding query while keeping key and value shared. It produces the same result as multi-head attention with less GPU memory and faster inference speed. We conduct extensive experiments on Transformer, BART, and GPT-2 for summarization and question generation tasks. The results show EL-attention speeds up existing models by 1.6x to 5.3x without accuracy loss.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 28 of its 36 references that are in the map. First posted on arXiv 11 May 2021.

Builds on

28 of 36 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  9. 2020Reformer: The Efficient Transformercited by 1,658
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
16 more
  1. 2015Teaching Machines to Read and Comprehendcited by 1,039
  2. 2018Image Transformercited by 770
  3. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  4. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  5. 2017Non-Autoregressive Neural Machine Translationcited by 457
  6. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  7. 2020Sparse Sinkhorn Attentioncited by 250
  8. 2018Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinementcited by 194
  9. 2020Low-Rank Bottleneck in Multi-head Attention Modelscited by 88
  10. 2018Tensor2Tensor for Neural Machine Translationcited by 85
  11. 2020Pre-trained Summarization Distillationcited by 70
  12. 2020X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformerscited by 58
  13. 2019Benchmarking TPU, GPU, and CPU Platforms for Deep Learningcited by 51
  14. 2020GLGE: A New General Language Generation Evaluation Benchmarkcited by 47
  15. 2020BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretrainingcited by 33
  16. 2020Hierarchical Roofline Performance Analysis for Deep Learning Applicationscited by 11

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 0; 2023: 1; 2024: 1; 2025: 3.

6 papers.

  1. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  2. 2025Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generationcited by 36
  3. 2021FastSeq: Make Sequence Generation Fastercited by 13
  4. 2025XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerializationcited by 4
  5. 2025Deconstructing Attention: Investigating Design Principles for Effective Language Modelingcited by 2
  6. 2023Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attentioncited by 1