# Numerical reasoning in machine reading comprehension tasks: are we there yet?

arXiv: https://arxiv.org/abs/2109.08207 · HTML: https://mlsanity.com/paper/2109.08207/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 26 references that are in the map.
- First posted on arXiv 16 Sep 2021.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [Injecting Numerical Reasoning Skills into Language Models](https://mlsanity.com/paper/2004.04487/) (2020)
- [How Can We Accelerate Progress Towards Human-like Linguistic Generalization?](https://mlsanity.com/paper/2005.00955/) (2020)
- [Representing Numbers in NLP: a Survey and a Vision](https://mlsanity.com/paper/2103.13136/) (2021)
- [Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?](https://mlsanity.com/paper/2012.15180/) (2020)
- [UnNatural Language Inference](https://mlsanity.com/paper/2101.00010/) (2020)
- [NumNet: Machine Reading Comprehension with Numerical Reasoning](https://mlsanity.com/paper/1910.06701/) (2019)
- [ExplainaBoard: An Explainable Leaderboard for NLP](https://mlsanity.com/paper/2104.06387/) (2021)
- [BERT & Family Eat Word Salad: Experiments with Text Understanding](https://mlsanity.com/paper/2101.03453/) (2021)
- [A Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoning](https://mlsanity.com/paper/1908.05514/) (2019)
- [EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference](https://mlsanity.com/paper/1901.03735/) (2019)
- [Diversify Your Datasets: Analyzing Generalization via Controlled Variance in Adversarial Datasets](https://mlsanity.com/paper/1910.09302/) (2019)
- [A Simple and Effective Model for Answering Multi-span Questions](https://mlsanity.com/paper/1909.13375/) (2019)
- [Semantics Altering Modifications for Evaluating Comprehension in Machine Reading](https://mlsanity.com/paper/2012.04056/) (2020)
- [An Empirical Study of Using Pre-trained BERT Models for Vietnamese Relation Extraction Task at VLSP 2020](https://mlsanity.com/paper/2012.10275/) (2020)

## Built on by (most cited first, 9 of 9)

- [Do Deep Neural Networks Capture Compositionality in Arithmetic Reasoning?](https://mlsanity.com/paper/2302.07866/) (2023)
- [Exploring the Landscape of Natural Language Processing Research](https://mlsanity.com/paper/2307.10652/) (2023)
- [Towards Robust Numerical Question Answering: Diagnosing Numerical Capabilities of NLP Systems](https://mlsanity.com/paper/2211.07455/) (2022)
- [A Survey on Measuring and Mitigating Reasoning Shortcuts in Machine Reading Comprehension](https://mlsanity.com/paper/2209.01824/) (2022)
- [Question Answering with LLMs and Learning from Answer Sets](https://mlsanity.com/paper/2509.16590/) (2025)
- [Towards preserving word order importance through Forced Invalidation](https://mlsanity.com/paper/2304.05221/) (2023)
- [ChemAlgebra: Algebraic Reasoning on Chemical Reactions](https://mlsanity.com/paper/2210.02095/) (2022)
- [How Well Do Multi-hop Reading Comprehension Models Understand Date Information?](https://mlsanity.com/paper/2210.05208/) (2022)
- [Empirical Investigation of Neural Symbolic Reasoning Strategies](https://mlsanity.com/paper/2302.08148/) (2023)

## Abstract (arXiv)

Numerical reasoning based machine reading comprehension is a task that involves reading comprehension along with using arithmetic operations such as addition, subtraction, sorting, and counting. The DROP benchmark (Dua et al., 2019) is a recent dataset that has inspired the design of NLP models aimed at solving this task. The current standings of these models in the DROP leaderboard, over standard metrics, suggest that the models have achieved near-human performance. However, does this mean that these models have learned to reason? In this paper, we present a controlled study on some of the top-performing model architectures for the task of numerical reasoning. Our observations suggest that the standard metrics are incapable of measuring progress towards such tasks.