# NT5?! Training T5 to Perform Numerical Reasoning

arXiv: https://arxiv.org/abs/2104.07307 · HTML: https://mlsanity.com/paper/2104.07307/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 9 references that are in the map.
- First posted on arXiv 15 Apr 2021.

## Builds on (most cited first, 9 of 9)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [Do NLP Models Know Numbers? Probing Numeracy in Embeddings](https://mlsanity.com/paper/1909.07940/) (2019)
- [Injecting Numerical Reasoning Skills into Language Models](https://mlsanity.com/paper/2004.04487/) (2020)
- [NumNet: Machine Reading Comprehension with Numerical Reasoning](https://mlsanity.com/paper/1910.06701/) (2019)
- [Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension](https://mlsanity.com/paper/1909.00109/) (2019)
- [EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference](https://mlsanity.com/paper/1901.03735/) (2019)
- [Question Directed Graph Attention Network for Numerical Reasoning over Text](https://mlsanity.com/paper/2009.07448/) (2020)

## Built on by (most cited first, 11 of 11)

- [Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills](https://mlsanity.com/paper/2107.07261/) (2021)
- [Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data](https://mlsanity.com/paper/2311.02216/) (2023)
- [QuanTemp: A real-world open-domain benchmark for fact-checking numerical claims](https://mlsanity.com/paper/2403.17169/) (2024)
- [Can neural networks do arithmetic? A survey on the elementary numerical skills of state-of-the-art deep learning models](https://mlsanity.com/paper/2303.07735/) (2023)
- [Investigating Numeracy Learning Ability of a Text-to-Text Transfer Model](https://mlsanity.com/paper/2109.04672/) (2021)
- [Do Deep Neural Networks Capture Compositionality in Arithmetic Reasoning?](https://mlsanity.com/paper/2302.07866/) (2023)
- [World Models for Math Story Problems](https://mlsanity.com/paper/2306.04347/) (2023)
- [Teaching Broad Reasoning Skills for Multi-Step QA by Generating Hard Contexts](https://mlsanity.com/paper/2205.12496/) (2022)
- [FERMAT: An Alternative to Accuracy for Numerical Reasoning](https://mlsanity.com/paper/2305.17491/) (2023)
- [Do Language Models Understand Measurements?](https://mlsanity.com/paper/2210.12694/) (2022)
- [Extracting Victim Counts from Text](https://mlsanity.com/paper/2302.12367/) (2023)

## Abstract (arXiv)

Numerical reasoning over text (NRoT) presents unique challenges that are not well addressed by existing pre-training objectives. We explore five sequential training schedules that adapt a pre-trained T5 model for NRoT. Our final model is adapted from T5, but further pre-trained on three datasets designed to strengthen skills necessary for NRoT and general reading comprehension before being fine-tuned on the Discrete Reasoning over Text (DROP) dataset. The training improves DROP's adjusted F1 performance (a numeracy-focused score) from 45.90 to 70.83. Our model closes in on GenBERT (72.4), a custom BERT-Base model using the same datasets with significantly more parameters. We show that training the T5 multitasking framework with multiple numerical reasoning datasets of increasing difficulty, good performance on DROP can be achieved without manually engineering partitioned functionality between distributed and symbol modules.