# Do Language Models Understand Measurements?

arXiv: https://arxiv.org/abs/2210.12694 · HTML: https://mlsanity.com/paper/2210.12694/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 23 Oct 2022.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [oLMpics -- On what Language Model Pre-training Captures](https://mlsanity.com/paper/1912.13283/) (2019)
- [Do NLP Models Know Numbers? Probing Numeracy in Embeddings](https://mlsanity.com/paper/1909.07940/) (2019)
- [Injecting Numerical Reasoning Skills into Language Models](https://mlsanity.com/paper/2004.04487/) (2020)
- [Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models](https://mlsanity.com/paper/2005.00683/) (2020)
- [Do Language Embeddings Capture Scales?](https://mlsanity.com/paper/2010.05345/) (2020)
- [Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numbers](https://mlsanity.com/paper/1805.08154/) (2018)
- [An Empirical Study of Multi-Task Learning on BERT for Biomedical Text Mining](https://mlsanity.com/paper/2005.02799/) (2020)
- [Verb Physics: Relative Physical Knowledge of Actions and Objects](https://mlsanity.com/paper/1706.03799/) (2017)
- [How Large Are Lions? Inducing Distributions over Quantitative Attributes](https://mlsanity.com/paper/1906.01327/) (2019)
- [NumGPT: Improving Numeracy Ability of Generative Pre-trained Models](https://mlsanity.com/paper/2109.03137/) (2021)
- [Investigating Numeracy Learning Ability of a Text-to-Text Transfer Model](https://mlsanity.com/paper/2109.04672/) (2021)
- [NT5?! Training T5 to Perform Numerical Reasoning](https://mlsanity.com/paper/2104.07307/) (2021)
- [Probing for Multilingual Numerical Understanding in Transformer-Based Language Models](https://mlsanity.com/paper/2010.06666/) (2020)

## Built on by (most cited first, 2 of 2)

- [Benchmarks as Microscopes: A Call for Model Metrology](https://mlsanity.com/paper/2407.16711/) (2024)
- [Application Of Vision-Language Models For Assessing Osteoarthritis Disease Severity](https://mlsanity.com/paper/2401.06331/) (2024)

## Abstract (arXiv)

Recent success of pre-trained language models (PLMs) has stimulated interest in their ability to understand and work with numbers. Yet, the numerical reasoning over measurements has not been formally studied despite their importance. In this study, we show that PLMs lack the capability required for reasoning over measurements. Furthermore, we find that a language model trained on a measurement-rich corpus shows better performance on understanding measurements. We propose a simple embedding strategy to better distinguish between numbers and units, which leads to a significant improvement in the probing tasks.