mlsanityData to 7 Oct 2026

QuALITY: Question Answering with Long Input Texts, Yes!

arXiv 2112.08608 ↗First posted 16 Dec 2021cs.CLcited by 204

Not on the map. Why?

Cited by 36 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

To enable building and testing models on long-document comprehension, we introduce QuALITY, a multiple-choice QA dataset with context passages in English that have an average length of about 5,000 tokens, much longer than typical current models can process. Unlike in prior work with passages, our questions are written and validated by contributors who have read the entire passage, rather than relying on summaries or excerpts. In addition, only half of the questions are answerable by annotators working under tight time constraints, indicating that skimming and simple search are not enough to consistently perform well. Our baseline models perform poorly on this task (55.4%) and significantly lag behind human performance (93.5%).

Key facts. Cited by 204 papers in the map (data through 7 Oct 2026). Builds on 35 of its 44 references that are in the map. First posted on arXiv 16 Dec 2021.

Builds on

35 of 44 references.

References · 35

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  9. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  10. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  11. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  12. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
23 more
  1. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  2. 2019ELI5: Long Form Question Answeringcited by 567
  3. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 409
  4. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  5. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  6. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  7. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
  8. 2017SearchQA: A New Q&A Dataset Augmented with Context from a Search Enginecited by 221
  9. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  10. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
  11. 2017Constructing Datasets for Multi-hop Reading Comprehension Across Documentscited by 190
  12. 2021Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answeringcited by 178
  13. 2020Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehensioncited by 131
  14. 2020Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?cited by 127
  15. 2015The Goldilocks Principle: Reading Children's Books with Explicit Memory Representationscited by 108
  16. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  17. 2019Compositional Questions Do Not Necessitate Multi-hop Reasoningcited by 87
  18. 2019Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QAcited by 63
  19. 2019Controlled Crowdsourcing for High-Quality QA-SRL Annotationcited by 34
  20. 2017Making Neural QA as Simple as Possible but not Simplercited by 33
  21. 2021Quiz-Style Question Generation for News Storiescited by 23
  22. 2021What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks?cited by 23
  23. 2016Embracing data abundance: BookTest Dataset for Reading Comprehensioncited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 19; 2023: 26; 2024: 52; 2025: 55; 2026: 52.

204 papers.

Sort:

By area:

  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  3. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  4. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  5. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  6. 2022UL2: Unifying Language Learning Paradigmscited by 292
  7. 2022Teaching language models to support answers with verified quotescited by 290
  8. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  9. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  10. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  11. 2025Survey on Evaluation of LLM-based Agentscited by 190
  12. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
38 more
  1. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  2. 2024A Human-Inspired Reading Agent with Gist Memory of Very Long Contextscited by 140
  3. 2024LLM Critics Help Catch LLM Bugscited by 138
  4. 2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 129
  5. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  6. 2023Scaling Laws of RoPE-based Extrapolationcited by 124
  7. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
  8. 2023AutoMix: Automatically Mixing Language Modelscited by 116
  9. 2023Retrieval meets Long Context Large Language Modelscited by 114
  10. 2023Scaling Transformer to 1M tokens and beyond with RMTcited by 97
  11. 2024On scalable oversight with weak LLMs judging strong LLMscited by 94
  12. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  13. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  14. 2025T*: Re-thinking Temporal Search for Long-Form Video Understandingcited by 82
  15. 2024Prover-Verifier Games improve legibility of LLM outputscited by 79
  16. 2024From Isolated Conversations to Hierarchical Schemas: Dynamic Tree Memory Representation for LLMscited by 77
  17. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 73
  18. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  19. 2022Efficient Long-Text Understanding with Short-Text Modelscited by 71
  20. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 65
  21. 2023PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documentscited by 61
  22. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 61
  23. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 60
  24. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 56
  25. 2024Synthetic continued pretrainingcited by 56
  26. 2023BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Modelscited by 53
  27. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  28. 2022SQuALITY: Building a Long-Document Summarization Dataset the Hard Waycited by 50
  29. 2024Evaluating LLMs at Detecting Errors in LLM Responsescited by 49
  30. 2026AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applicationscited by 46
  31. 2025In-depth Analysis of Graph-based RAG in a Unified Frameworkcited by 45
  32. 2022Token Dropping for Efficient BERT Pretrainingcited by 42
  33. 2024UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysiscited by 40
  34. 2025Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMscited by 40
  35. 2022An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classificationcited by 38
  36. 2024DOCBENCH: A Benchmark for Evaluating LLM-based Document Reading Systemscited by 37
  37. 2024On the Structural Memory of LLM Agentscited by 37
  38. 2024How Susceptible are LLMs to Influence in Prompts?cited by 32

50 most cited of 204 All 204 (JSON)