mlsanityData to 7 Oct 2026

Limitations of Language Models in Arithmetic and Symbolic Induction

arXiv 2208.05051 ↗First posted 9 Aug 2022cs.CLcited by 71

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Recent work has shown that large pretrained Language Models (LMs) can not only perform remarkably well on a range of Natural Language Processing (NLP) tasks but also start improving on reasoning tasks such as arithmetic induction, symbolic manipulation, and commonsense reasoning with increasing size of models. However, it is still unclear what the underlying capabilities of these LMs are. Surprisingly, we find that these models have limitations on certain basic symbolic manipulation tasks such as copy, reverse, and addition. When the total number of symbols or repeating symbols increases, the model performance drops quickly. We investigate the potential causes behind this phenomenon and examine a set of possible methods, including explicit positional markers, fine-grained computation steps, and LMs with callable programs. Experimental results show that none of these techniques can solve the simplest addition induction problem completely. In the end, we introduce LMs with tutor, which demonstrates every single step of teaching. LMs with tutor is able to deliver 100% accuracy in situations of OOD and repeating symbols, shedding new insights on the boundary of large LMs in induction.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 15 of its 23 references that are in the map. First posted on arXiv 9 Aug 2022.

Builds on

15 of 23 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Solving Quantitative Reasoning Problems with Language Models29 Jun

Other references · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  8. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  9. 2015Pointer Networkscited by 882
  10. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  11. 2014Neural Turing Machinescited by 630
  12. 2016Solving General Arithmetic Word Problemscited by 353
2 more
  1. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 136
  2. 2021Teaching Autoregressive Language Models Complex Tasks By Demonstrationcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 28; 2024: 26; 2025: 11; 2026: 4.

71 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2022PAL: Program-aided Language Modelscited by 635
  3. 2023Augmented Language Models: a Surveycited by 398
  4. 2023Faithful Chain-of-Thought Reasoningcited by 285
  5. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  6. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  7. 2023Challenges and Applications of Large Language Modelscited by 251
  8. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  9. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 165
  10. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  11. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 129
  12. 2023Teaching Arithmetic to Small Transformerscited by 121
38 more
  1. 2023Large Language Models are Few-Shot Health Learnerscited by 115
  2. 2024Chain of Thoughtlessness? An Analysis of CoT in Planningcited by 100
  3. 2023Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Taskscited by 84
  4. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  5. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 77
  6. 2023Auto-Regressive Next-Token Predictors are Universal Learnerscited by 65
  7. 2023Positional Description Matters for Transformers Arithmeticcited by 56
  8. 2024Making Pre-trained Language Models Great on Tabular Predictioncited by 56
  9. 2023Autonomous GIS: the next-generation AI-powered GIScited by 52
  10. 2023Large Language Models for Compiler Optimizationcited by 52
  11. 2023Evaluating and Improving Tool-Augmented Computation-Intensive Math Reasoningcited by 49
  12. 2024Interpreting and Improving Large Language Models in Arithmetic Calculationcited by 43
  13. 2024Pre-trained Large Language Models Use Fourier Features to Compute Additioncited by 38
  14. 2024How Likely Do LLMs with CoT Mimic Human Reasoning?cited by 34
  15. 2023Evidence of interrelated cognitive-like capabilities in large language models: Indications of artificial general intelligence or achievement?cited by 26
  16. 2023Bot or Human? Detecting ChatGPT Imposters with A Single Questioncited by 25
  17. 2023GPT as Knowledge Worker: A Zero-Shot Evaluation of (AI)CPA Capabilitiescited by 22
  18. 2024Large Language Models Lack Understanding of Character Composition of Wordscited by 21
  19. 2024See it, Think it, Sorted: Large Multimodal Models are Few-shot Time Series Anomaly Analyzerscited by 20
  20. 2024Verbalized Machine Learning: Revisiting Machine Learning with Language Modelscited by 17
  21. 2024Planning in Strawberry Fields: Evaluating and Improving the Planning and Scheduling Capabilities of LRM o1cited by 17
  22. 2025The Lookahead Limitation: Why Multi-Operand Addition is Hard for LLMscited by 13
  23. 2023Mathematics, word problems, common sense, and artificial intelligencecited by 12
  24. 2023Mastering Symbolic Operations: Augmenting Language Models with Compiled Neural Networkscited by 12
  25. 2024Fine-Tuning on Diverse Reasoning Chains Drives Within-Inference CoT Refinement in LLMscited by 12
  26. 2023Do Deep Neural Networks Capture Compositionality in Arithmetic Reasoning?cited by 11
  27. 2023Synergistic Integration of Large Language Models and Cognitive Architectures for Robust AI: An Exploratory Analysiscited by 11
  28. 2024A Theory for Length Generalization in Learning to Reasoncited by 11
  29. 2024Your Context Is Not an Array: Unveiling Random Access Limitations in Transformerscited by 11
  30. 2023From Good to Great: Improving Math Reasoning with Tool-Augmented Interleaf Promptingcited by 10
  31. 2024Reverse That Number! Decoding Order Matters in Arithmetic Learningcited by 8
  32. 2023Conditions for Length Generalization in Learning Reasoning Skillscited by 7
  33. 2024Dissecting Multiplication in Transformers: Insights into LLMscited by 7
  34. 2023A Comprehensive Evaluation of Tool-Assisted Generation Strategiescited by 6
  35. 2024RNR: Teaching Large Language Models to Follow Roles and Rulescited by 6
  36. 2023STEPS: A Benchmark for Order Reasoning in Sequential Taskscited by 5
  37. 2024Retrieved In-Context Principles from Previous Mistakescited by 5
  38. 2024Principled Understanding of Generalization for Generative Transformer Models in Arithmetic Reasoning Taskscited by 5

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Investigating Symbolic Capabilities of Large Language Models
  2. 2023Frugal LMs Trained to Invoke Symbolic Solvers Achieve Parameter-Efficient Arithmetic Reasoning
  3. 2025Do Large Language Models Truly Grasp Addition? A Rule-Focused Diagnostic Using Two-Integer Arithmetic
  4. 2024Unraveling Arithmetic in Large Language Models: The Role of Algebraic Structures
  5. 2023Reasoning in Large Language Models Through Symbolic Math Word Problems
  6. 2024Assessing the Emergent Symbolic Reasoning Abilities of Llama Large Language Models
  7. 2024Language Models Do Hard Arithmetic Tasks Easily and Hardly Do Easy Arithmetic Tasks
  8. 2023Controlling Equational Reasoning in Large Language Models with Prompt Interventions