mlsanityData to 7 Oct 2026

Numerical reasoning in machine reading comprehension tasks: are we there yet?

arXiv 2109.08207 ↗First posted 16 Sep 2021cs.CL cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Numerical reasoning based machine reading comprehension is a task that involves reading comprehension along with using arithmetic operations such as addition, subtraction, sorting, and counting. The DROP benchmark (Dua et al., 2019) is a recent dataset that has inspired the design of NLP models aimed at solving this task. The current standings of these models in the DROP leaderboard, over standard metrics, suggest that the models have achieved near-human performance. However, does this mean that these models have learned to reason? In this paper, we present a controlled study on some of the top-performing model architectures for the task of numerical reasoning. Our observations suggest that the standard metrics are incapable of measuring progress towards such tasks.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 21 of its 26 references that are in the map. First posted on arXiv 16 Sep 2021.

Builds on

21 of 26 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  5. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  6. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  7. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  8. 2020Injecting Numerical Reasoning Skills into Language Modelscited by 133
  9. 2020How Can We Accelerate Progress Towards Human-like Linguistic Generalization?cited by 114
  10. 2021Representing Numbers in NLP: a Survey and a Visioncited by 83
  11. 2020Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?cited by 72
  12. 2020UnNatural Language Inferencecited by 68
9 more
  1. 2019NumNet: Machine Reading Comprehension with Numerical Reasoningcited by 59
  2. 2021ExplainaBoard: An Explainable Leaderboard for NLPcited by 50
  3. 2021BERT & Family Eat Word Salad: Experiments with Text Understandingcited by 44
  4. 2019A Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoningcited by 41
  5. 2019EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inferencecited by 39
  6. 2019Diversify Your Datasets: Analyzing Generalization via Controlled Variance in Adversarial Datasetscited by 21
  7. 2019A Simple and Effective Model for Answering Multi-span Questionscited by 13
  8. 2020Semantics Altering Modifications for Evaluating Comprehension in Machine Readingcited by 13
  9. 2020An Empirical Study of Using Pre-trained BERT Models for Vietnamese Relation Extraction Task at VLSP 2020cited by 1

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 4; 2024: 0; 2025: 1.

9 papers.

  1. 2023Do Deep Neural Networks Capture Compositionality in Arithmetic Reasoning?cited by 11
  2. 2023Exploring the Landscape of Natural Language Processing Researchcited by 7
  3. 2022Towards Robust Numerical Question Answering: Diagnosing Numerical Capabilities of NLP Systemscited by 6
  4. 2022A Survey on Measuring and Mitigating Reasoning Shortcuts in Machine Reading Comprehensioncited by 4
  5. 2025Question Answering with LLMs and Learning from Answer Setscited by 4
  6. 2023Towards preserving word order importance through Forced Invalidationcited by 2
  7. 2022ChemAlgebra: Algebraic Reasoning on Chemical Reactionscited by 1
  8. 2022How Well Do Multi-hop Reading Comprehension Models Understand Date Information?cited by 1
  9. 2023Empirical Investigation of Neural Symbolic Reasoning Strategiescited by 1