# NUMCoT: Numerals and Units of Measurement in Chain-of-Thought Reasoning using Large Language Models

arXiv: https://arxiv.org/abs/2406.02864 · HTML: https://mlsanity.com/paper/2406.02864/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 21 of its 28 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [Solving General Arithmetic Word Problems](https://mlsanity.com/paper/1608.01413/) (2016)
- [Do NLP Models Know Numbers? Probing Numeracy in Embeddings](https://mlsanity.com/paper/1909.07940/) (2019)
- [NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks](https://mlsanity.com/paper/2204.05660/) (2022)
- [Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models](https://mlsanity.com/paper/2005.00683/) (2020)
- [On the Computational Power of Transformers and its Implications in Sequence Modeling](https://mlsanity.com/paper/2006.09286/) (2020)
- [Representing Numbers in NLP: a Survey and a Vision](https://mlsanity.com/paper/2103.13136/) (2021)
- [GPT Can Solve Mathematical Problems Without a Calculator](https://mlsanity.com/paper/2309.03241/) (2023)
- [LSTM Networks Can Perform Dynamic Counting](https://mlsanity.com/paper/1906.03648/) (2019)
- [Verb Physics: Relative Physical Knowledge of Actions and Objects](https://mlsanity.com/paper/1706.03799/) (2017)
- [How Large Are Lions? Inducing Distributions over Quantitative Attributes](https://mlsanity.com/paper/1906.01327/) (2019)
- [Investigating Math Word Problems using Pretrained Multilingual Language Models](https://mlsanity.com/paper/2105.08928/) (2021)
- [Exploring Generalization Ability of Pretrained Language Models on Arithmetic and Logical Reasoning](https://mlsanity.com/paper/2108.06743/) (2021)
- [Towards Question Format Independent Numerical Reasoning: A Set of Prerequisite Tasks](https://mlsanity.com/paper/2005.08516/) (2020)
- [Probing for Multilingual Numerical Understanding in Transformer-Based Language Models](https://mlsanity.com/paper/2010.06666/) (2020)

## Abstract (arXiv)

Numeral systems and units of measurement are two conjoined topics in activities of human beings and have mutual effects with the languages expressing them. Currently, the evaluation of Large Language Models (LLMs) often involves mathematical reasoning, yet little attention is given to how minor changes in numbers or units can drastically alter the complexity of problems and the performance of LLMs. In this paper, we scrutinize existing LLMs on processing of numerals and units of measurement by constructing datasets with perturbations. We first anatomize the reasoning of math word problems to different sub-procedures like numeral conversions from language to numbers and measurement conversions based on units. Then we further annotate math word problems from ancient Chinese arithmetic works which are challenging in numerals and units of measurement. Experiments on perturbed datasets demonstrate that LLMs still encounter difficulties in handling numeral and measurement conversions.