mlsanityData to 7 Oct 2026

ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding

arXiv 2305.14196 ↗First posted 23 May 2023cs.CL cs.AI cs.LGcited by 175

Not on the map. Why?

Cited by 22 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce ZeroSCROLLS, a zero-shot benchmark for natural language understanding over long texts, which contains only test and small validation sets, without training data. We adapt six tasks from the SCROLLS benchmark, and add four new datasets, including two novel information fusing tasks, such as aggregating the percentage of positive reviews. Using ZeroSCROLLS, we conduct a comprehensive evaluation of both open-source and closed large language models, finding that Claude outperforms ChatGPT, and that GPT-4 achieves the highest average score. However, there is still room for improvement on multiple open challenges in ZeroSCROLLS, such as aggregation tasks, where models struggle to pass the naive baseline. As the state of the art is a moving target, we invite researchers to evaluate their ideas on the live ZeroSCROLLS leaderboard.

Key facts. Cited by 175 papers in the map (data through 7 Oct 2026). Builds on 29 of its 36 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

29 of 36 references.

References · 29

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  9. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  10. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  11. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  12. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
17 more
  1. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  2. 2021Efficient Attentions for Long Document Summarizationcited by 342
  3. 2022UL2: Unifying Language Learning Paradigmscited by 292
  4. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  5. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  6. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  7. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  8. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  9. 2020Few-Shot Text Generation with Pattern-Exploiting Trainingcited by 87
  10. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  11. 2020The Turking Test: Can Language Models Understand Instructions?cited by 82
  12. 2021SummScreen: A Dataset for Abstractive Screenplay Summarizationcited by 80
  13. 2020Extractive Opinion Summarization in Quantized Transformer Spacescited by 76
  14. 2022Efficient Long-Text Understanding with Short-Text Modelscited by 71
  15. 2022SQuALITY: Building a Long-Document Summarization Dataset the Hard Waycited by 50
  16. 2022Investigating Efficiently Extending Transformers for Long Input Summarizationcited by 43
  17. 2022Adapting Pretrained Text-to-Text Models for Long Text Sequencescited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 16; 2024: 76; 2025: 47; 2026: 36.

175 papers.

Sort:

By area:

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  4. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  5. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  6. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  7. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
  8. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  9. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  10. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  11. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  12. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
38 more
  1. 2023Universal Self-Consistency for Large Language Model Generationcited by 198
  2. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  3. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  4. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 150
  5. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  6. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  7. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  8. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  9. 2024Transformers are Multi-State RNNscited by 120
  10. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
  11. 2024Make Your LLM Fully Utilize the Contextcited by 118
  12. 2023AutoMix: Automatically Mixing Language Modelscited by 116
  13. 2024AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformerscited by 116
  14. 2023Retrieval meets Long Context Large Language Modelscited by 114
  15. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  16. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Taskscited by 94
  17. 2024Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systemscited by 87
  18. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  19. 2024Self-Organized Agents: A LLM Multi-Agent Framework toward Ultra Large-Scale Code Generation and Optimizationcited by 84
  20. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 80
  21. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  22. 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
  23. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 68
  24. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 57
  25. 2023BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Modelscited by 53
  26. 2024Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inferencecited by 51
  27. 2024LongGenBench: Benchmarking Long-Form Generation in Long Context LLMscited by 47
  28. 2025The Amazon Nova Family of Models: Technical Report and Model Cardcited by 41
  29. 2024In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Misscited by 36
  30. 2024Insights into LLM Long-Context Failures: When Transformers Know but Don't Tellcited by 36
  31. 2024RepoQA: Evaluating Long Context Code Understandingcited by 35
  32. 2025Oolong: Evaluating Long Context Reasoning and Aggregation Capabilitiescited by 33
  33. 2023Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encodingcited by 32
  34. 2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 32
  35. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32
  36. 2024Long2RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recallcited by 32
  37. 2024NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokenscited by 31
  38. 2024LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarioscited by 31

50 most cited of 175 All 175 (JSON)