mlsanityData to 14 Oct 2026

The Geometry of Numerical Reasoning: Language Models Compare Numeric Properties in Linear Subspaces

arXiv 2410.13194 ↗First posted 17 Oct 2024cs.CLcited by 10

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

This paper investigates whether large language models (LLMs) utilize numerical attributes encoded in a low-dimensional subspace of the embedding space when answering questions involving numeric comparisons, e.g., Was Cristiano born before Messi? We first identified, using partial least squares regression, these subspaces, which effectively encode the numerical attributes associated with the entities in comparison prompts. Further, we demonstrate causality, by intervening in these subspaces to manipulate hidden states, thereby altering the LLM's comparison outcomes. Experiments conducted on three different LLMs showed that our results hold across different numerical attributes, indicating that LLMs utilize the linearly encoded information for numerical reasoning.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 17 of its 25 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

17 of 25 references.

References · 17

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  5. 2018Deep contextualized word representationscited by 2,290
  6. 2019Language Models as Knowledge Bases?cited by 1,936
  7. 2019How Can We Know What Language Models Know?cited by 959
  8. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  9. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  10. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 200
  11. 2019Do NLP Models Know Numbers? Probing Numeracy in Embeddingscited by 159
  12. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
5 more
  1. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  2. 2020Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queriescited by 107
  3. 2024What does the Knowledge Neuron Thesis Have to do with Knowledge?cited by 49
  4. 2024Interpreting and Improving Large Language Models in Arithmetic Calculationcited by 45
  5. 2018Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numberscited by 42

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2025Hypothesis-Driven Feature Manifold Analysis in LLMs via Supervised Multi-Dimensional Scalingcited by 8
  2. 2025Number Representations in LLMs: A Computational Parallel to Human Perceptioncited by 4
  3. 2025Can Language Models Handle a Non-Gregorian Calendar? The Case of the Japanese warekicited by 1
  4. 2026What Really Controls Temporal Reasoning in Large Language Models: Tokenisation or Representation of Time?cited by 1
  5. 2026Tool-Call Dependency Structure is Linearly Decodable in LLM Agent Residual Streamscited by 1
  6. 2026The Count Is There, but Misaligned: Understanding and Correcting Counting Failures in VLMscited by 1
  7. 2025Interpreting Multi-Attribute Confounding through Numerical Attributes in Large Language Modelscited by 0
  8. 2026DriveCode: Domain Specific Numerical Encoding for LLM-Based Autonomous Drivingcited by 0
  9. 2026Language Models Compare Quantities Using Number-specific and Unit-specific Heuristicscited by 0
  10. 2026When Models Don't Manipulate Manifolds: The Geometry of a Comparison Taskcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Reflection of Thought: Inversely Eliciting Numerical Reasoning in Language Models via Solving Linear Systems
  2. 2023Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data
  3. 2025Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Models
  4. 2024Language Models Encode the Value of Numbers Linearly
  5. 2024NumeroLogic: Number Encoding for Enhanced LLMs' Numerical Reasoning
  6. 2026The Effect of Scripts and Formats on LLM Numeracy
  7. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis
  8. 2026ANI: Adaptive Numerical Injection for Unifying Semantic and Arithmetic Representations in Numerical Reasoning