mlsanityData to 7 Oct 2026

Teaching Algorithmic Reasoning via In-context Learning

arXiv 2211.09066 ↗First posted 15 Nov 2022cs.LG cs.AI cs.CLcited by 111

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Large language models (LLMs) have shown increasing in-context learning capabilities through scaling up model and data size. Despite this progress, LLMs are still unable to solve algorithmic reasoning problems. While providing a rationale with the final answer has led to further improvements in multi-step reasoning problems, Anil et al. 2022 showed that even simple algorithmic reasoning tasks such as parity are far from solved. In this work, we identify and study four key stages for successfully teaching algorithmic reasoning to LLMs: (1) formulating algorithms as skills, (2) teaching multiple skills simultaneously (skill accumulation), (3) teaching how to combine skills (skill composition) and (4) teaching how to use skills as tools. We show that it is possible to teach algorithmic reasoning to LLMs via in-context learning, which we refer to as algorithmic prompting. We evaluate our approach on a variety of arithmetic and quantitative reasoning tasks, and demonstrate significant boosts in performance over existing prompting techniques. In particular, for long parity, addition, multiplication and subtraction, we achieve an error reduction of approximately 10x, 9x, 5x and 2x respectively compared to the best available baselines.

Key facts. Cited by 111 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 71 references that are in the map. First posted on arXiv 15 Nov 2022.

Builds on

43 of 71 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Solving Quantitative Reasoning Problems with Language Models29 Jun
  2. 2020Generative Language Modeling for Automated Theorem Proving7 Sep

Other references · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  11. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  12. 2022Emergent Abilities of Large Language Modelscited by 2,483
29 more
  1. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  2. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  3. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  4. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  5. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  6. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  7. 2022Autoformalization with Large Language Modelscited by 252
  8. 2022Exploring Length Generalization in Large Language Modelscited by 204
  9. 2022TALM: Tool Augmented Language Modelscited by 197
  10. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanationscited by 159
  11. 2019What Can Neural Networks Reason About?cited by 155
  12. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  13. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 136
  14. 2022Faithful Reasoning Using Large Language Modelscited by 130
  15. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tangocited by 128
  16. 2015Neural GPUs Learn Algorithmscited by 118
  17. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
  18. 2022Rationale-Augmented Ensembles in Language Modelscited by 111
  19. 2022Capturing Failures of Large Language Models via Human Cognitive Biasescited by 97
  20. 2022Thor: Wielding Hammers to Integrate Language Models and Automated Theorem Proverscited by 97
  21. 2021Neural Algorithmic Reasoningcited by 92
  22. 2021Representing Numbers in NLP: a Survey and a Visioncited by 83
  23. 2020Compositional Generalization via Neural-Symbolic Stack Machinescited by 80
  24. 2021NaturalProofs: Mathematical Theorem Proving in Natural Languagecited by 80
  25. 2018Semantically-Aligned Equation Generation for Solving and Reasoning Math Word Problemscited by 49
  26. 2020Neural Execution Engines: Learning to Execute Subroutinescited by 36
  27. 2021How to transfer algorithmic reasoning knowledge to learn new algorithms?cited by 22
  28. 2019Do Multi-hop Readers Dream of Reasoning Chains?cited by 14
  29. 2021Towards Tractable Mathematical Reasoning: Challenges, Strategies, and Opportunities for Solving Math Word Problemscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 48; 2024: 29; 2025: 21; 2026: 6.

111 papers.

Sort:

By area:

  1. 2023Teaching Large Language Models to Self-Debugcited by 963
  2. 2022A Survey on In-context Learningcited by 717
  3. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  4. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  5. 2023Llemma: An Open Language Model For Mathematicscited by 385
  6. 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 298
  7. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  8. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  9. 2023Can Language Models Solve Graph Problems in Natural Language?cited by 249
  10. 2023Looped Transformers as Programmable Computerscited by 224
  11. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 135
  12. 2023Deductive Verification of Chain-of-Thought Reasoningcited by 122
38 more
  1. 2023Teaching Arithmetic to Small Transformerscited by 121
  2. 2023Hypothesis Search: Inductive Reasoning with Language Modelscited by 119
  3. 2024Tokenization counts: the impact of tokenization on arithmetic in frontier LLMscited by 112
  4. 2024Chain of Thoughtlessness? An Analysis of CoT in Planningcited by 100
  5. 2023Amortizing intractable inference in large language modelscited by 94
  6. 2025DeepSeek-R1 Thoughtology: Let's think about LLM Reasoningcited by 91
  7. 2023Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Taskscited by 84
  8. 2024Transformers Can Achieve Length Generalization But Not Robustlycited by 82
  9. 2023Few-shot In-context Learning for Knowledge Base Question Answeringcited by 74
  10. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 74
  11. 2023Evaluating Large Language Models on Controlled Generation Taskscited by 73
  12. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  13. 2022Complementary Explanations for Effective In-Context Learningcited by 71
  14. 2023GPT Can Solve Mathematical Problems Without a Calculatorcited by 68
  15. 2022LAMBADA: Backward Chaining for Automated Reasoning in Natural Languagecited by 66
  16. 2023Large Language Models can Learn Rulescited by 66
  17. 2023Length Generalization in Arithmetic Transformerscited by 64
  18. 2024Understanding Transformer Reasoning Capabilities via Graph Algorithmscited by 64
  19. 2022Parsel: Algorithmic Reasoning with Language Models by Composing Decompositionscited by 62
  20. 2023Towards Better Chain-of-Thought Prompting Strategies: A Surveycited by 62
  21. 2023Prompted LLMs as Chatbot Modules for Long Open-domain Conversationcited by 61
  22. 2023Positional Description Matters for Transformers Arithmeticcited by 56
  23. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  24. 2024Why Larger Language Models Do In-context Learning Differently?cited by 44
  25. 2023Self-Polish: Enhance Reasoning in Large Language Models via Problem Refinementcited by 43
  26. 2024Evolving Code with A Large Language Modelcited by 42
  27. 2023Code Execution with Pre-trained Language Modelscited by 33
  28. 2023Learning the greatest common divisor: explaining transformer predictionscited by 33
  29. 2024Leveraging the Context through Multi-Round Interactions for Jailbreaking Attackscited by 32
  30. 2024The CLRS-Text Algorithmic Reasoning Language Benchmarkcited by 28
  31. 2024Language Models Need Inductive Biases to Count Inductivelycited by 25
  32. 2024In-Context Learning Enables Robot Action Prediction in LLMscited by 25
  33. 2023Mathematical Language Models: A Surveycited by 24
  34. 2025Social Genome: Grounded Social Reasoning Abilities of Multimodal Modelscited by 24
  35. 2024How Well Can Transformers Emulate In-context Newton's Method?cited by 23
  36. 2023Passive learning of active causal strategies in agents and language modelscited by 21
  37. 2023Language models are weak learnerscited by 21
  38. 2024Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descentcited by 21

50 most cited of 111 All 111 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Benchmarking Large Language Models for Math Reasoning Tasks
  2. 2024Multi-Step Reasoning with Large Language Models, a Survey
  3. 2024Teaching-Inspired Integrated Prompting Framework: A Novel Approach for Enhancing Reasoning in Large Language Models
  4. 2025Teaching LLMs According to Their Aptitude: Adaptive Reasoning for Mathematical Problem Solving
  5. 2025A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematics
  6. 2026Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges
  7. 2024From Next-Token to Mathematics: The Learning Dynamics of Mathematical Reasoning in Language Models
  8. 2024What Makes In-context Learning Effective for Mathematical Reasoning: A Theoretical Analysis