# Interpreting and Improving Large Language Models in Arithmetic Calculation

arXiv: https://arxiv.org/abs/2409.01659 · HTML: https://mlsanity.com/paper/2409.01659/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 63 references that are in the map.
- First posted on arXiv 3 Sep 2024.

## Builds on (most cited first, 30 of 45)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [Solving math word problems with process- and outcome-based feedback](https://mlsanity.com/paper/2211.14275/) (2022)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Causal Abstractions of Neural Networks](https://mlsanity.com/paper/2106.02997/) (2021)
- [Analysing Mathematical Reasoning Abilities of Neural Models](https://mlsanity.com/paper/1904.01557/) (2019)
- [Direct and Indirect Effects](https://mlsanity.com/paper/1301.2300/) (2013)
- [Mathematical Capabilities of ChatGPT](https://mlsanity.com/paper/2301.13867/) (2023)
- [MathPrompter: Mathematical Reasoning using Large Language Models](https://mlsanity.com/paper/2303.05398/) (2023)
- [Making Large Language Models Better Reasoners with Step-Aware Verifier](https://mlsanity.com/paper/2206.02336/) (2022)
- [How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model](https://mlsanity.com/paper/2305.00586/) (2023)
- [Localizing Model Behavior with Path Patching](https://mlsanity.com/paper/2304.05969/) (2023)
- [Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit](https://mlsanity.com/paper/2207.08799/) (2022)

## Built on by (most cited first, 30 of 45)

- [Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics](https://mlsanity.com/paper/2410.21272/) (2024)
- [MIB: A Mechanistic Interpretability Benchmark](https://mlsanity.com/paper/2504.13151/) (2025)
- [Number Cookbook: Number Understanding of Language Models and How to Improve It](https://mlsanity.com/paper/2411.03766/) (2024)
- [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026)
- [Probing for Arithmetic Errors in Language Models](https://mlsanity.com/paper/2507.12379/) (2025)
- [Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism](https://mlsanity.com/paper/2504.18574/) (2025)
- [Implicit Reasoning in Transformers is Reasoning through Shortcuts](https://mlsanity.com/paper/2503.07604/) (2025)
- [Language Models are Symbolic Learners in Arithmetic](https://mlsanity.com/paper/2410.15580/) (2024)
- [The Geometry of Numerical Reasoning: Language Models Compare Numeric Properties in Linear Subspaces](https://mlsanity.com/paper/2410.13194/) (2024)
- [Math Neurosurgery: Isolating Language Models' Math Reasoning Abilities Using Only Forward Passes](https://mlsanity.com/paper/2410.16930/) (2024)
- [The Validation Gap: A Mechanistic Analysis of How Language Models Compute Arithmetic but Fail to Validate It](https://mlsanity.com/paper/2502.11771/) (2025)
- [Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts](https://mlsanity.com/paper/2605.01148/) (2026)
- [Process or Result? Manipulated Ending Tokens Can Mislead Reasoning LLMs to Ignore the Correct Reasoning Steps](https://mlsanity.com/paper/2503.19326/) (2025)
- [Understanding In-context Learning of Addition via Activation Subspaces](https://mlsanity.com/paper/2505.05145/) (2025)
- [Can LLMs Reason Abstractly Over Math Word Problems Without CoT? Disentangling Abstract Formulation From Arithmetic Computation](https://mlsanity.com/paper/2505.23701/) (2025)
- [Efficient numeracy in language models through single-token number embeddings](https://mlsanity.com/paper/2510.06824/) (2025)
- [Benford's Curse: Tracing Digit Bias to Numerical Hallucination in LLMs](https://mlsanity.com/paper/2506.01734/) (2025)
- [Modular Arithmetic: Language Models Solve Math Digit by Digit](https://mlsanity.com/paper/2508.02513/) (2025)
- [Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models](https://mlsanity.com/paper/2504.15471/) (2025)
- [All for One: LLMs Solve Mental Math at the Last Token With Information Transferred From Other Tokens](https://mlsanity.com/paper/2509.09650/) (2025)
- [Unraveling Arithmetic in Large Language Models: The Role of Algebraic Structures](https://mlsanity.com/paper/2411.16260/) (2024)
- [Fine, I'll Merge It Myself: A Multi-Fidelity Framework for Automated Model Merging](https://mlsanity.com/paper/2502.04030/) (2025)
- [Leveraging Submodule Linearity Enhances Task Arithmetic Performance in LLMs](https://mlsanity.com/paper/2504.10902/) (2025)
- [Exploring the Translation Mechanism of Large Language Models](https://mlsanity.com/paper/2502.11806/) (2025)
- [Exploring the Hidden Reasoning Process of Large Language Models by Misleading Them](https://mlsanity.com/paper/2503.16401/) (2025)
- [Restoring Pruned Large Language Models via Lost Component Compensation](https://mlsanity.com/paper/2510.21834/) (2025)
- [Shattered Compositionality: Counterintuitive Learning Dynamics of Transformers for Arithmetic](https://mlsanity.com/paper/2601.22510/) (2026)
- [Interpreting the Latent Structure of Operator Precedence in Language Models](https://mlsanity.com/paper/2510.13908/) (2025)
- [Language Models Learn Universal Representations of Numbers and Here's Why You Should Care](https://mlsanity.com/paper/2510.26285/) (2025)
- [Compressed Sensing for Capability Localization in Large Language Models](https://mlsanity.com/paper/2603.03335/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable potential across numerous applications and have shown an emergent ability to tackle complex reasoning tasks, such as mathematical computations. However, even for the simplest arithmetic calculations, the intrinsic mechanisms behind LLMs remain mysterious, making it challenging to ensure reliability. In this work, we delve into uncovering a specific mechanism by which LLMs execute calculations. Through comprehensive experiments, we find that LLMs frequently involve a small fraction (< 5%) of attention heads, which play a pivotal role in focusing on operands and operators during calculation processes. Subsequently, the information from these operands is processed through multi-layer perceptrons (MLPs), progressively leading to the final solution. These pivotal heads/MLPs, though identified on a specific dataset, exhibit transferability across different datasets and even distinct tasks. This insight prompted us to investigate the potential benefits of selectively fine-tuning these essential heads/MLPs to boost the LLMs' computational performance. We empirically find that such precise tuning can yield notable enhancements on mathematical prowess, without compromising the performance on non-mathematical tasks. Our work serves as a preliminary exploration into the arithmetic calculation abilities inherent in LLMs, laying a solid foundation to reveal more intricate mathematical tasks.