mlsanityData to 7 Oct 2026

The NLP Task Effectiveness of Long-Range Transformers

arXiv 2202.07856 ↗First posted 16 Feb 2022cs.CL cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

Transformer models cannot easily scale to long sequences due to their O(N^2) time and space complexity. This has led to Transformer variants seeking to lower computational complexity, such as Longformer and Performer. While such models have theoretically greater efficiency, their effectiveness on real NLP tasks has not been well studied. We benchmark 7 variants of Transformer models on 5 difficult NLP tasks and 7 datasets. We design experiments to isolate the effect of pretraining and hyperparameter settings, to focus on their capacity for long-range attention. Moreover, we present various methods to investigate attention behaviors to illuminate model details beyond metric scores. We find that the modified attention in long-range transformers has advantages on content selection and query-guided decoding, but they come with previously unrecognized drawbacks such as insufficient attention to distant tokens and accumulated approximation error.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 38 of its 50 references that are in the map. First posted on arXiv 16 Feb 2022.

Builds on

38 of 50 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  11. 2020Reformer: The Efficient Transformercited by 1,658
  12. 2020Rethinking Attention with Performerscited by 1,584
26 more
  1. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  2. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  3. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  4. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  5. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  6. 2020Efficient Transformers: A Surveycited by 784
  7. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  8. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  9. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  10. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  11. 2021Random Feature Attentioncited by 350
  12. 2018AllenNLP: A Deep Semantic Natural Language Processing Platformcited by 347
  13. 2021Efficient Attentions for Long Document Summarizationcited by 342
  14. 2020Sparse Sinkhorn Attentioncited by 250
  15. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  16. 2015Visualizing and Understanding Neural Models in NLPcited by 199
  17. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  18. 2018Higher-order Coreference Resolution with Coarse-to-fine Inferencecited by 150
  19. 2019BERT for Coreference Resolution: Baselines and Analysiscited by 125
  20. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  21. 2021SummScreen: A Dataset for Abstractive Screenplay Summarizationcited by 80
  22. 2021DocNLI: A Large-scale Dataset for Document-level Natural Language Inferencecited by 75
  23. 2021Coreference Resolution without Span Representationscited by 45
  24. 2021ABC: Attention with Bounded-memory Controlcited by 36
  25. 2020Do Transformers Need Deep Long-Range Memorycited by 24
  26. 2021Database Reasoning Over Textcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 5; 2024: 6; 2025: 6; 2026: 3.

22 papers.

Sort:

By area:

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  2. 2022UL2: Unifying Language Learning Paradigmscited by 292
  3. 2023Personality Traits in Large Language Modelscited by 166
  4. 2023Domain-specific Continued Pretraining of Language Models for Capturing Long Context in Mental Healthcited by 25
  5. 2023Dodo: Dynamic Contextual Compression for Decoder-only LMscited by 23
  6. 2023Nugget: Neural Agglomerative Embeddings of Textcited by 19
  7. 2024CLERC: A Dataset for Legal Case Retrieval and Retrieval-Augmented Analysis Generationcited by 18
  8. 2025Lost at the Beginning of Reasoningcited by 15
  9. 2024Multilingual Needle in a Haystack: Investigating Long-Context Behavior of Multilingual Large Language Modelscited by 9
  10. 2025Masks Can Be Distracting: On Context Comprehension in Diffusion Language Modelscited by 7
  11. 2024Attention Instruction: Amplifying Attention in the Middle via Promptingcited by 6
  12. 2024Length-Aware Multi-Kernel Transformer for Long Document Classificationcited by 5
10 more
  1. 2024Are queries and keys always relevant? A case study on Transformer wave functionscited by 5
  2. 2024CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modelingcited by 4
  3. 2025Temporal Relation Extraction in Clinical Texts: A Span-based Graph Transformer Approachcited by 4
  4. 2025AgentFlux: Decoupled Fine-Tuning & Inference for On-Device Agentic Systemscited by 2
  5. 2025LLP: LLM-Based Product Pricing in E-commercecited by 2
  6. 2026CoMI-IRL: Contrastive Multi-Intention Inverse Reinforcement Learningcited by 2
  7. 2026Training and Evaluating Diffusion Policies with Long Context Lengthscited by 2
  8. 2022Memory Augmented Lookup Dictionary based Language Modeling for Automatic Speech Recognitioncited by 1
  9. 2025Path Pooling: Training-Free Structure Enhancement for Efficient Knowledge Graph Retrieval-Augmented Generationcited by 0
  10. 2026Contextualized Early Detection of Online Firestorms: A Sequential LLM-Based Approachcited by 0