mlsanityData to 7 Oct 2026

Big Bird: Transformers for Longer Sequences

arXiv 2007.14062 ↗First posted 28 Jul 2020cs.LG cs.CL stat.MLcited by 1,673

Not on the map. Why?

Cited by 94 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformers-based models, such as BERT, have been one of the most successful deep learning models for NLP. Unfortunately, one of their core limitations is the quadratic dependency (mainly in terms of memory) on the sequence length due to their full attention mechanism. To remedy this, we propose, BigBird, a sparse attention mechanism that reduces this quadratic dependency to linear. We show that BigBird is a universal approximator of sequence functions and is Turing complete, thereby preserving these properties of the quadratic, full attention model. Along the way, our theoretical analysis reveals some of the benefits of having O(1) global tokens (such as CLS), that attend to the entire sequence as part of the sparse attention mechanism. The proposed sparse attention can handle sequences of length up to 8x of what was previously possible using similar hardware. As a consequence of the capability to handle longer context, BigBird drastically improves performance on various NLP tasks such as question answering and summarization. We also propose novel applications to genomics data.

Key facts. Cited by 1,673 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 202; 2022: 269; 2023: 296; 2024: 283; 2025: 298; 2026: 325.

1,673 papers.

Sort:

By area:

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  6. 2023A Survey of Large Language Modelscited by 2,537
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  9. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  11. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,200
  12. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
38 more
  1. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  2. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  3. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformerscited by 707
  4. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
  5. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  6. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
  7. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  8. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  9. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  10. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  11. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  12. 2021A Survey of Transformerscited by 399
  13. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  14. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  15. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  16. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  17. 2021Random Feature Attentioncited by 350
  18. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  19. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  20. 2021Efficient Attentions for Long Document Summarizationcited by 342
  21. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  22. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  23. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 302
  24. 2022AudioGen: Textually Guided Audio Generationcited by 292
  25. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 286
  26. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  27. 2022Long Range Language Modeling via Gated State Spacescited by 265
  28. 2021Generating Images with Sparse Representationscited by 262
  29. 2021Measuring Attribution in Natural Language Generation Modelscited by 253
  30. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 253
  31. 2023A Cookbook of Self-Supervised Learningcited by 248
  32. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  33. 2021Voxel Transformer for 3D Object Detectioncited by 247
  34. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  35. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  36. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  37. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  38. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229

50 most cited of 1,673 All 1,673 (JSON)