mlsanityData to 7 Oct 2026

How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model

arXiv 2305.00586 ↗First posted 30 Apr 2023cs.CL cs.AI cs.LGcited by 199

Not on the map. Why?

Cited by 20 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Pre-trained language models can be surprisingly adept at tasks they were not explicitly trained on, but how they implement these capabilities is poorly understood. In this paper, we investigate the basic mathematical abilities often acquired by pre-trained language models. Concretely, we use mechanistic interpretability techniques to explain the (limited) mathematical abilities of GPT-2 small. As a case study, we examine its ability to take in sentences such as "The war lasted from the year 1732 to the year 17", and predict valid two-digit end years (years > 32). We first identify a circuit, a small subset of GPT-2 small's computational graph that computes this task's output. Then, we explain the role of each circuit component, showing that GPT-2 small's final multi-layer perceptrons boost the probability of end years greater than the start year. Finally, we find related tasks that activate our circuit. Our results suggest that GPT-2 small computes greater-than using a complex but general mechanism that activates across diverse contexts.

Key facts. Cited by 199 papers in the map (data through 7 Oct 2026). Builds on 38 of its 53 references that are in the map. First posted on arXiv 30 Apr 2023.

Builds on

38 of 53 references.

References · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  4. 2022Emergent Abilities of Large Language Modelscited by 2,483
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  7. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  8. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  9. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  10. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  11. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  12. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
26 more
  1. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  2. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  3. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  4. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  5. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  6. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  7. 2021Causal Abstractions of Neural Networkscited by 401
  8. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
  9. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  10. 2023Localizing Model Behavior with Path Patchingcited by 182
  11. 2019Do NLP Models Know Numbers? Probing Numeracy in Embeddingscited by 157
  12. 2020Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?cited by 144
  13. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 136
  14. 2020Injecting Numerical Reasoning Skills into Language Modelscited by 133
  15. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  16. 2018What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Modelscited by 124
  17. 2018Identifying and Controlling Important Neurons in Neural Machine Translationcited by 110
  18. 2022NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Taskscited by 110
  19. 2018Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Informationcited by 97
  20. 2020Analyzing Individual Neurons in Pre-trained Language Modelscited by 97
  21. 2019The emergence of number and syntax units in LSTM language modelscited by 92
  22. 2021Representing Numbers in NLP: a Survey and a Visioncited by 83
  23. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  24. 2022Limitations of Language Models in Arithmetic and Symbolic Inductioncited by 71
  25. 2020How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Maskingcited by 63
  26. 2021Not All Models Localize Linguistic Knowledge in the Same Place: A Layer-wise Probing on BERToids' Representationscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 19; 2024: 77; 2025: 56; 2026: 47.

199 papers.

Sort:

By area:

  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  2. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  3. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
  4. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  5. 2023Language Models Represent Space and Timecited by 306
  6. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
  7. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 201
  8. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
  9. 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 196
  10. 2024How to use and interpret activation patchingcited by 191
  11. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 186
  12. 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 144
38 more
  1. 2025Thought Anchors: Which LLM Reasoning Steps Matter?cited by 126
  2. 2023Teaching Arithmetic to Small Transformerscited by 121
  3. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  4. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 111
  5. 2023Circuit Component Reuse Across Tasks in Transformer Language Modelscited by 105
  6. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 100
  7. 2024ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretabilitycited by 100
  8. 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristicscited by 94
  9. 2024Information Flow Routes: Automatically Interpreting Language Models at Scalecited by 92
  10. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  11. 2024Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Modelscited by 87
  12. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  13. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 78
  14. 2023Successor Heads: Recurring, Interpretable Attention Heads In The Wildcited by 77
  15. 2023Copy Suppression: Comprehensively Understanding an Attention Headcited by 73
  16. 2024Attention Heads of Large Language Models: A Surveycited by 65
  17. 2024LLM Circuit Analyses Are Consistent Across Training and Scalecited by 60
  18. 2024Finding Transformer Circuits with Edge Pruningcited by 58
  19. 2024Confidence Regulation Neurons in Language Modelscited by 57
  20. 2023Positional Description Matters for Transformers Arithmeticcited by 56
  21. 2024Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thoughtcited by 56
  22. 2024From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuningcited by 51
  23. 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neuronscited by 47
  24. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  25. 2024Interpreting Attention Layer Outputs with Sparse Autoencoderscited by 43
  26. 2024Interpreting and Improving Large Language Models in Arithmetic Calculationcited by 43
  27. 2026When Models Manipulate Manifolds: The Geometry of a Counting Taskcited by 43
  28. 2024Softmax is not Enough (for Sharp Size Generalisation)cited by 42
  29. 2024Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactualscited by 40
  30. 2024Talking Heads: Understanding Inter-layer Communication in Transformer Language Modelscited by 39
  31. 2024Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficientcited by 39
  32. 2024Pre-trained Large Language Models Use Fourier Features to Compute Additioncited by 38
  33. 2023Interpretable Diffusion via Information Decompositioncited by 37
  34. 2024IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representationscited by 37
  35. 2024Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questionscited by 35
  36. 2025Detection and Mitigation of Hallucination in Large Reasoning Models: A Mechanistic Perspectivecited by 32
  37. 2024Iteration Head: A Mechanistic Study of Chain-of-Thoughtcited by 31
  38. 2024UniBias: Unveiling and Mitigating LLM Bias through Internal Attention and FFN Manipulationcited by 30

50 most cited of 199 All 199 (JSON)