mlsanityData to 14 Oct 2026

Show Your Work: Scratchpads for Intermediate Computation with Language Models

arXiv 2112.00114 ↗First posted 30 Nov 2021cs.LG cs.NEcited by 886

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large pre-trained language models perform remarkably well on tasks that can be done "in one pass", such as generating realistic text or synthesizing computer programs. However, they struggle with tasks that require unbounded multi-step computation, such as adding integers or executing programs. Surprisingly, we find that these same models are able to perform complex multi-step computations -- even in the few-shot regime -- when asked to perform the operation "step by step", showing the results of intermediate computations. In particular, we train transformers to perform multi-step computations by asking them to emit intermediate computation steps into a "scratchpad". On a series of increasingly complex tasks ranging from long addition to the execution of arbitrary programs, we show that scratchpads dramatically improve the ability of language models to perform multi-step computations.

Key facts. Cited by 886 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 24 of its 35 references that are in the map. First posted on arXiv 30 Nov 2021.

Builds on

24 of 35 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  7. 2018Universal Transformerscited by 647
  8. 2014Neural Turing Machinescited by 644
  9. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 506
  10. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 335
  11. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  12. 2017Learning to Represent Programs with Graphscited by 220
12 more
  1. 2021Implicit Representations of Meaning in Neural Language Modelscited by 180
  2. 2021PonderNet: Learning to Pondercited by 168
  3. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  4. 2017RobustFill: Neural Program Learning under Noisy I/Ocited by 126
  5. 2015Neural GPUs Learn Algorithmscited by 121
  6. 2019Neural Execution of Graph Algorithmscited by 121
  7. 2014Learning to Executecited by 106
  8. 2015Neural Programmer-Interpreterscited by 101
  9. 2021Neural Algorithmic Reasoningcited by 94
  10. 2020Learning to Execute Programs with Instruction Pointer Attention Graph Neural Networkscited by 35
  11. 2015Neural Random-Access Machinescited by 34
  12. 2020Learning Semantic Program Embeddings with Graph Interval Neural Networkcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 106; 2023: 231; 2024: 187; 2025: 214; 2026: 147.

886 papers.

Sort:

By area:

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2023A Survey of Large Language Modelscited by 2,576
  11. 2022Emergent Abilities of Large Language Modelscited by 2,512
  12. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
38 more
  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  2. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  3. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  4. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  5. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  6. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  8. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  9. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  10. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  11. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  12. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  13. 2023Teaching Large Language Models to Self-Debugcited by 988
  14. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  15. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  16. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  17. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  18. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  19. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  20. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  21. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  22. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  23. 2022PAL: Program-aided Language Modelscited by 652
  24. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  25. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 574
  26. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  27. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 551
  28. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 548
  29. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  30. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  31. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 500
  32. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  33. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  34. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 431
  35. 2023Cognitive Architectures for Language Agentscited by 429
  36. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  37. 2023Augmented Language Models: a Surveycited by 403
  38. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 384

50 most cited of 886 All 886 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023ART: Automatic multi-step reasoning and tool-use for large language models
  2. 2026Not Too Hard, Not Too Easy: Learning from Intermediate States for LLM Structured Reasoning
  3. 2024Assessing Code Generation with Intermediate Languages
  4. 2023Specializing Smaller Language Models towards Multi-Step Reasoning
  5. 2026What do Language Models Learn and When? The Implicit Curriculum Hypothesis
  6. 2026Exploring the Limits of Pruning: Task-Specific Neurons, Model Collapse, and Recovery in Task-Specific Large Language Models
  7. 2024Cross-model Control: Improving Multiple Large Language Models in One-time Training
  8. 2023L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models