# The Geometry of Numerical Reasoning: Language Models Compare Numeric Properties in Linear Subspaces

arXiv: https://arxiv.org/abs/2410.13194 · HTML: https://mlsanity.com/paper/2410.13194/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 17 of its 25 references that are in the map.
- First posted on arXiv 17 Oct 2024.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Linearity of Relation Decoding in Transformer Language Models](https://mlsanity.com/paper/2308.09124/) (2023)
- [Do NLP Models Know Numbers? Probing Numeracy in Embeddings](https://mlsanity.com/paper/1909.07940/) (2019)
- [Probing Pretrained Language Models for Lexical Semantics](https://mlsanity.com/paper/2010.05731/) (2020)
- [Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models](https://mlsanity.com/paper/2102.00894/) (2021)
- [Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries](https://mlsanity.com/paper/2008.09036/) (2020)
- [What does the Knowledge Neuron Thesis Have to do with Knowledge?](https://mlsanity.com/paper/2405.02421/) (2024)
- [Interpreting and Improving Large Language Models in Arithmetic Calculation](https://mlsanity.com/paper/2409.01659/) (2024)
- [Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numbers](https://mlsanity.com/paper/1805.08154/) (2018)

## Built on by (most cited first, 10 of 10)

- [Hypothesis-Driven Feature Manifold Analysis in LLMs via Supervised Multi-Dimensional Scaling](https://mlsanity.com/paper/2510.01025/) (2025)
- [Number Representations in LLMs: A Computational Parallel to Human Perception](https://mlsanity.com/paper/2502.16147/) (2025)
- [Can Language Models Handle a Non-Gregorian Calendar? The Case of the Japanese wareki](https://mlsanity.com/paper/2509.04432/) (2025)
- [What Really Controls Temporal Reasoning in Large Language Models: Tokenisation or Representation of Time?](https://mlsanity.com/paper/2603.19017/) (2026)
- [Tool-Call Dependency Structure is Linearly Decodable in LLM Agent Residual Streams](https://mlsanity.com/paper/2605.25310/) (2026)
- [The Count Is There, but Misaligned: Understanding and Correcting Counting Failures in VLMs](https://mlsanity.com/paper/2607.09544/) (2026)
- [Interpreting Multi-Attribute Confounding through Numerical Attributes in Large Language Models](https://mlsanity.com/paper/2511.04053/) (2025)
- [DriveCode: Domain Specific Numerical Encoding for LLM-Based Autonomous Driving](https://mlsanity.com/paper/2603.00919/) (2026)
- [Language Models Compare Quantities Using Number-specific and Unit-specific Heuristics](https://mlsanity.com/paper/2606.03982/) (2026)
- [When Models Don't Manipulate Manifolds: The Geometry of a Comparison Task](https://mlsanity.com/paper/2609.37680/) (2026)

## Abstract (arXiv)

This paper investigates whether large language models (LLMs) utilize numerical attributes encoded in a low-dimensional subspace of the embedding space when answering questions involving numeric comparisons, e.g., Was Cristiano born before Messi? We first identified, using partial least squares regression, these subspaces, which effectively encode the numerical attributes associated with the entities in comparison prompts. Further, we demonstrate causality, by intervening in these subspaces to manipulate hidden states, thereby altering the LLM's comparison outcomes. Experiments conducted on three different LLMs showed that our results hold across different numerical attributes, indicating that LLMs utilize the linearly encoded information for numerical reasoning.