mlsanityData to 7 Oct 2026

LazyFormer: Self Attention with Lazy Update

arXiv 2102.12702 ↗First posted 25 Feb 2021cs.CL cs.AIcited by 16

Not on the map. Why?

Abstract (arXiv)

Improving the efficiency of Transformer-based language pre-training is an important task in NLP, especially for the self-attention module, which is computationally expensive. In this paper, we propose a simple but effective solution, called LazyFormer, which computes the self-attention distribution infrequently. LazyFormer composes of multiple lazy blocks, each of which contains multiple Transformer layers. In each lazy block, the self-attention distribution is only computed once in the first layer and then is reused in all upper layers. In this way, the cost of computation could be largely saved. We also provide several training tricks for LazyFormer. Extensive experiments demonstrate the effectiveness of the proposed method.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 20 of its 25 references that are in the map. First posted on arXiv 25 Feb 2021.

Builds on

20 of 25 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  11. 2020Reformer: The Efficient Transformercited by 1,658
  12. 2020Rethinking Attention with Performerscited by 1,584
8 more
  1. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  2. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  3. 2017Mixed Precision Trainingcited by 1,135
  4. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  5. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  6. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  7. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
  8. 2019Sharing Attention Weights for Fast Transformercited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 1; 2023: 3; 2024: 4; 2025: 3; 2026: 2.

16 papers.

Sort:

By area:

  1. 2021A Survey of Transformerscited by 399
  2. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 318
  3. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharingcited by 38
  4. 2023Skip-Attention: Improving Vision Transformers by Paying Less Attentioncited by 23
  5. 2021LightSeq2: Accelerated Training for Transformer-based Models on GPUscited by 19
  6. 2022CAB: Comprehensive Attention Benchmarking on Long Sequence Modelingcited by 10
  7. 2025Kascade: A Practical Sparse Attention Method for Long-Context LLM Inferencecited by 9
  8. 2024Head-wise Shareable Attention for Large Language Modelscited by 4
  9. 2024Adaptive Patching for High-resolution Image Segmentation with Transformerscited by 4
  10. 2023Exploring Attention Map Reuse for Efficient Transformer Neural Networkscited by 3
  11. 2023Ultra-Long Sequence Distributed Transformercited by 3
  12. 2024EchoAtt: Attend, Copy, then Adjust for More Efficient Large Language Modelscited by 2
4 more
  1. 2025AttnCache: Accelerating Self-Attention Inference for LLM Prefill via Attention Cachecited by 2
  2. 2026HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computingcited by 2
  3. 2025Scalable Spatiotemporal Inference with Biased Scan Attention Transformer Neural Processescited by 1
  4. 2026EPAS: Efficient Training with Progressive Activation Sharingcited by 0