mlsanityData to 7 Oct 2026

Longformer: The Long-Document Transformer

arXiv 2004.05150 ↗First posted 10 Apr 2020cs.CLcited by 3,362

Not on the map. Why?

Cited by 196 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer. Longformer's attention mechanism is a drop-in replacement for the standard self-attention and combines a local windowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we evaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In contrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on WikiHop and TriviaQA. We finally introduce the Longformer-Encoder-Decoder (LED), a Longformer variant for supporting long document generative sequence-to-sequence tasks, and demonstrate its effectiveness on the arXiv summarization dataset.

Key facts. Cited by 3,362 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 331; 2022: 446; 2023: 569; 2024: 725; 2025: 688; 2026: 602.

3,362 papers.

Sort:

By area:

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  3. 2023Qwen Technical Reportcited by 3,273
  4. 2023Mistral 7Bcited by 3,065
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  9. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  10. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  11. 2025Gemma 3 Technical Reportcited by 1,686
  12. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
38 more
  1. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  2. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  3. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,200
  4. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  5. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  6. 2021Transformers in Vision: A Surveycited by 1,009
  7. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  8. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  9. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  10. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  11. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  12. 2020Aligning AI With Shared Human Valuescited by 706
  13. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
  14. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  15. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  16. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 656
  17. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
  18. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  19. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  20. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  21. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  22. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 519
  23. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  24. 2024Large Language Models: A Surveycited by 506
  25. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  26. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  27. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  28. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  29. 2021A Survey of Transformerscited by 399
  30. 2022Multimodal Learning with Transformers: A Surveycited by 399
  31. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  32. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  33. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  34. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  35. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  36. 2021Random Feature Attentioncited by 350
  37. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  38. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346

50 most cited of 3,362 All 3,362 (JSON)