# Can Large Language Models put 2 and 2 together? Probing for Entailed Arithmetical Relationships

arXiv: https://arxiv.org/abs/2404.19432 · HTML: https://mlsanity.com/paper/2404.19432/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 25 references that are in the map.
- First posted on arXiv 30 Apr 2024.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning](https://mlsanity.com/paper/2305.12295/) (2023)
- [Evaluating Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2310.19736/) (2023)
- [Logic Tensor Networks](https://mlsanity.com/paper/2012.13635/) (2020)
- [Can Large Language Models Infer Causation from Correlation?](https://mlsanity.com/paper/2306.05836/) (2023)
- [Injecting Numerical Reasoning Skills into Language Models](https://mlsanity.com/paper/2004.04487/) (2020)
- [Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models](https://mlsanity.com/paper/2005.00683/) (2020)
- [Coercing LLMs to do and reveal (almost) anything](https://mlsanity.com/paper/2402.14020/) (2024)
- [GLUCOSE: GeneraLized and COntextualized Story Explanations](https://mlsanity.com/paper/2009.07758/) (2020)
- [Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data](https://mlsanity.com/paper/2402.17644/) (2024)
- [Assessing Prompt Injection Risks in 200+ Custom GPTs](https://mlsanity.com/paper/2311.11538/) (2023)
- [How Likely Do LLMs with CoT Mimic Human Reasoning?](https://mlsanity.com/paper/2402.16048/) (2024)
- [Traces of Memorisation in Large Language Models for Code](https://mlsanity.com/paper/2312.11658/) (2023)
- [Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection](https://mlsanity.com/paper/2404.09894/) (2024)
- [Verb Physics: Relative Physical Knowledge of Actions and Objects](https://mlsanity.com/paper/1706.03799/) (2017)
- [LLMs for Relational Reasoning: How Far are We?](https://mlsanity.com/paper/2401.09042/) (2024)
- [Structured Learning Modulo Theories](https://mlsanity.com/paper/1405.1675/) (2014)

## Built on by (most cited first, 1 of 1)

- [Intermediate Languages Matter: Formal Languages and LLMs affect Neurosymbolic Reasoning](https://mlsanity.com/paper/2509.04083/) (2025)

## Abstract (arXiv)

Two major areas of interest in the era of Large Language Models regard questions of what do LLMs know, and if and how they may be able to reason (or rather, approximately reason). Since to date these lines of work progressed largely in parallel (with notable exceptions), we are interested in investigating the intersection: probing for reasoning about the implicitly-held knowledge. Suspecting the performance to be lacking in this area, we use a very simple set-up of comparisons between cardinalities associated with elements of various subjects (e.g. the number of legs a bird has versus the number of wheels on a tricycle). We empirically demonstrate that although LLMs make steady progress in knowledge acquisition and (pseudo)reasoning with each new GPT release, their capabilities are limited to statistical inference only. It is difficult to argue that pure statistical learning can cope with the combinatorial explosion inherent in many commonsense reasoning tasks, especially once arithmetical notions are involved. Further, we argue that bigger is not always better and chasing purely statistical improvements is flawed at the core, since it only exacerbates the dangerous conflation of the production of correct answers with genuine reasoning ability.