mlsanityData to 7 Oct 2026

DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs

arXiv 1903.00161 ↗First posted 1 Mar 2019cs.CLcited by 883

Not on the map. Why?

Cited by 140 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Reading comprehension has recently seen rapid progress, with systems matching humans on the most popular datasets for the task. However, a large body of work has highlighted the brittleness of these systems, showing that there is much work left to be done. We introduce a new English reading comprehension benchmark, DROP, which requires Discrete Reasoning Over the content of Paragraphs. In this crowdsourced, adversarially-created, 96k-question benchmark, a system must resolve references in a question, perhaps to multiple input positions, and perform discrete operations over them (such as addition, counting, or sorting). These operations require a much more comprehensive understanding of the content of paragraphs than what was necessary for prior datasets. We apply state-of-the-art methods from both the reading comprehension and semantic parsing literature on this dataset and show that the best systems only achieve 32.7% F1 on our generalized accuracy metric, while expert human performance is 96.0%. We additionally present a new model that combines reading comprehension methods with simple numerical reasoning to achieve 47.0% F1.

Key facts. Cited by 883 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 106; 2022: 101; 2023: 112; 2024: 150; 2025: 240; 2026: 174.

883 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  8. 2023A Survey of Large Language Modelscited by 2,537
  9. 2025Gemma 3 Technical Reportcited by 1,686
  10. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  11. 2023PaLM 2 Technical Reportcited by 1,154
  12. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
38 more
  1. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  2. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  3. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  5. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  6. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  7. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  8. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  9. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  10. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  11. 2022Large Language Models Can Self-Improvecited by 656
  12. 2023A Comprehensive Overview of Large Language Modelscited by 639
  13. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  14. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  15. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  16. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  17. 2025Kimi K2: Open Agentic Intelligencecited by 523
  18. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  19. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  20. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  21. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  22. 2025Humanity's Last Examcited by 467
  23. 2021MetaICL: Learning to Learn In Contextcited by 450
  24. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  25. 2021InfographicVQAcited by 430
  26. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  27. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
  28. 2024AFlow: Automating Agentic Workflow Generationcited by 395
  29. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  30. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  31. 2024Automated Design of Agentic Systemscited by 360
  32. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  33. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  34. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  35. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  36. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  37. 2022FOLIO: Natural Language Reasoning with First-Order Logiccited by 233
  38. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 231

50 most cited of 883 All 883 (JSON)