mlsanityData to 7 Oct 2026

Interpreting Arithmetic Mechanism in Large Language Models through Comparative Neuron Analysis

arXiv 2409.14144 ↗First posted 21 Sep 2024cs.CLcited by 31

Not on the map. Why?

Abstract (arXiv)

We find arithmetic ability resides within a limited number of attention heads, with each head specializing in distinct operations. To delve into the reason, we introduce the Comparative Neuron Analysis (CNA) method, which identifies an internal logic chain consisting of four distinct stages from input to prediction: feature enhancing with shallow FFN neurons, feature transferring by shallow attention layers, feature predicting by arithmetic heads, and prediction enhancing among deep FFN neurons. Moreover, we identify the human-interpretable FFN neurons within both feature-enhancing and feature-predicting stages. These findings lead us to investigate the mechanism of LoRA, revealing that it enhances prediction probabilities by amplifying the coefficient scores of FFN neurons related to predictions. Finally, we apply our method in model pruning for arithmetic tasks and model editing for reducing gender bias. Code is on https://github.com/zepingyu0512/arithmetic-mechanism.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 33 of its 43 references that are in the map. First posted on arXiv 21 Sep 2024.

Builds on

33 of 43 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  6. 2016Layer Normalizationcited by 5,579
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2017Axiomatic Attribution for Deep Networkscited by 3,350
21 more
  1. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  2. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  3. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  4. 2022Mass-Editing Memory in a Transformercited by 976
  5. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  6. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  7. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  8. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  9. 2023Llemma: An Open Language Model For Mathematicscited by 385
  10. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  11. 2013Direct and Indirect Effectscited by 276
  12. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  13. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
  14. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  15. 2023Successor Heads: Recurring, Interpretable Attention Heads In The Wildcited by 77
  16. 2021MathBERT: A Pre-Trained Model for Mathematical Formula Understandingcited by 58
  17. 2021Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Modelscited by 57
  18. 2022Discovering Language-neutral Sub-networks in Multilingual Language Modelscited by 28
  19. 2022What Are You Token About? Dense Retrieval as Distributions Over the Vocabularycited by 28
  20. 2024Do Language Models Exhibit the Same Cognitive Biases in Problem Solving as Human Learners?cited by 19
  21. 2024How do Large Language Models Learn In-Context? Query and Key Matrices of In-Context Heads are Two Towers for Metric Learningcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 19; 2026: 10.

31 papers.

Sort:

By area:

  1. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  2. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  3. 2024A Implies B: Circuit Analysis in LLMs for Propositional Logical Reasoningcited by 15
  4. 2025Understanding and Mitigating Gender Bias in LLMs via Interpretable Neuron Editingcited by 14
  5. 2025Implicit Reasoning in Transformers is Reasoning through Shortcutscited by 14
  6. 2025Understanding How Value Neurons Shape the Generation of Specified Values in LLMscited by 13
  7. 2024Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answeringcited by 11
  8. 2025Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language Modelscited by 8
  9. 2025Can LLMs Reason Abstractly Over Math Word Problems Without CoT? Disentangling Abstract Formulation From Arithmetic Computationcited by 8
  10. 2025Where Culture Fades: Revealing the Cultural Gap in Text-to-Image Generationcited by 8
  11. 2025Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMscited by 7
  12. 2025Leveraging the true depth of LLMscited by 6
19 more
  1. 2025Benford's Curse: Tracing Digit Bias to Numerical Hallucination in LLMscited by 6
  2. 2025Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Modelscited by 6
  3. 2025Benchmark Profiling: Mechanistic Diagnosis of LLM Benchmarkscited by 5
  4. 2025Brain-Inspired Exploration of Functional Networks and Key Neurons in Large Language Modelscited by 4
  5. 2026Deconstructing Pre-training: Knowledge Attribution Analysis in MoE and Dense Modelscited by 4
  6. 2025Addition in Four Movements: Mapping Layer-wise Information Trajectories in LLMscited by 3
  7. 2026NEAT: Neuron-Based Early Exit for Large Reasoning Modelscited by 3
  8. 2025DFAMS: Dynamic-flow guided Federated Alignment based Multi-prototype Searchcited by 2
  9. 2025StreetMath: Study of LLMs' Approximation Behaviorscited by 2
  10. 2025ClueAnchor: Clue-Anchored Knowledge Reasoning Exploration and Optimization for Retrieval-Augmented Generationcited by 1
  11. 2026How Do LLMs and VLMs Understand Viewpoint Rotation Without Vision? An Interpretability Studycited by 1
  12. 2026Tiny Brains, Giant Impact: Uncovering the Keystone Neurons of LLM with Just a Few Promptscited by 1
  13. 2025CircuitProbe: Tracing Visual Temporal Evidence Flow in Video Language Modelscited by 0
  14. 2025Two Heads are Better than One: Distilling Large Language Model Features Into Small Models with Feature Decomposition and Mixturecited by 0
  15. 2026Mechanistic Decoding of Cognitive Constructs in Large Language Modelscited by 0
  16. 2026Why Knowing Both Hops Is Not Enough: Understanding Two-Hop Generalization in Language Modelscited by 0
  17. 2026When Tokenization is Secretly Output Supervisioncited by 0
  18. 2026ViD: Vision-Dominant Gender Bias Mitigation for Large Vision-Language Modelscited by 0
  19. 2026Spontaneous Context Restoration: How Language Models Recover from Corrupted Inputscited by 0