mlsanityData to 7 Oct 2026

SCROLLS: Standardized CompaRison Over Long Language Sequences

arXiv 2201.03533 ↗First posted 10 Jan 2022cs.CL cs.AI cs.LGcited by 170

Not on the map. Why?

Abstract (arXiv)

NLP benchmarks have largely focused on short texts, such as sentences and paragraphs, even though long texts comprise a considerable amount of natural language in the wild. We introduce SCROLLS, a suite of tasks that require reasoning over long texts. We examine existing long-text datasets, and handpick ones where the text is naturally long, while prioritizing tasks that involve synthesizing information across the input. SCROLLS contains summarization, question answering, and natural language inference tasks, covering multiple domains, including literature, science, business, and entertainment. Initial baselines, including Longformer Encoder-Decoder, indicate that there is ample room for improvement on SCROLLS. We make all datasets available in a unified text-to-text format and host a live leaderboard to facilitate research on model architecture and pretraining methods.

Key facts. Cited by 170 papers in the map (data through 7 Oct 2026). Builds on 30 of its 46 references that are in the map. First posted on arXiv 10 Jan 2022.

Builds on

30 of 46 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  9. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  10. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  11. 2015Teaching Machines to Read and Comprehendcited by 1,039
  12. 2020Efficient Transformers: A Surveycited by 784
18 more
  1. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  2. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  3. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  4. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  5. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  6. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  7. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  8. 2021Random Feature Attentioncited by 350
  9. 2021Efficient Attentions for Long Document Summarizationcited by 342
  10. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  11. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  12. 2019MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehensioncited by 183
  13. 2021ContractNLI: A Dataset for Document-level Natural Language Inference for Contractscited by 145
  14. 2018Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Contextcited by 113
  15. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  16. 2021SummScreen: A Dataset for Abstractive Screenplay Summarizationcited by 80
  17. 2020Shortformer: Better Language Modeling using Shorter Inputscited by 76
  18. 2021A Practical Survey on Faster and Lighter Transformerscited by 73

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 24; 2023: 40; 2024: 51; 2025: 25; 2026: 30.

170 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  4. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  5. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  6. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  7. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  8. 2022UL2: Unifying Language Learning Paradigmscited by 292
  9. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255
  10. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  11. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  12. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
38 more
  1. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  2. 2024An Empirical Study of Mamba-based Language Modelscited by 177
  3. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  4. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  5. 2024A Human-Inspired Reading Agent with Gist Memory of Very Long Contextscited by 140
  6. 2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 129
  7. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  8. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  9. 2022A Length-Extrapolatable Transformercited by 123
  10. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  11. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  12. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
  13. 2023AutoMix: Automatically Mixing Language Modelscited by 116
  14. 2023Retrieval meets Long Context Large Language Modelscited by 114
  15. 2022Block-Recurrent Transformerscited by 107
  16. 2022What Makes Convolutional Models Great on Long Sequence Modeling?cited by 86
  17. 2024Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuningcited by 85
  18. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  19. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  20. 2022Parallel Context Windows for Large Language Modelscited by 81
  21. 2023In-context Pretraining: Language Modeling Beyond Document Boundariescited by 76
  22. 2022Efficient Transformers with Dynamic Token Poolingcited by 75
  23. 2022Efficient Long-Text Understanding with Short-Text Modelscited by 71
  24. 2023PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documentscited by 61
  25. 2026FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scalingcited by 58
  26. 2023BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Modelscited by 53
  27. 2024ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilitiescited by 52
  28. 2022Pretraining Without Attentioncited by 51
  29. 2024Linearizing Large Language Modelscited by 51
  30. 2022SQuALITY: Building a Long-Document Summarization Dataset the Hard Waycited by 50
  31. 2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 47
  32. 2024Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Lengthcited by 46
  33. 2023Extending Context Window of Large Language Models via Semantic Compressioncited by 45
  34. 2022Conditional Generation with a Question-Answering Blueprintcited by 44
  35. 2022Investigating Efficiently Extending Transformers for Long Input Summarizationcited by 43
  36. 2023When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Taskscited by 37
  37. 2024Insights into LLM Long-Context Failures: When Transformers Know but Don't Tellcited by 36
  38. 2024Suri: Multi-constraint Instruction Following for Long-form Text Generationcited by 34

50 most cited of 170 All 170 (JSON)