mlsanityData to 7 Oct 2026

Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks

arXiv 2211.12588 ↗First posted 22 Nov 2022cs.CL cs.AIcited by 1,057

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Recently, there has been significant progress in teaching language models to perform step-by-step reasoning to solve complex numerical reasoning tasks. Chain-of-thoughts prompting (CoT) is by far the state-of-art method for these tasks. CoT uses language models to perform both reasoning and computation in the multi-step `thought' process. To disentangle computation from reasoning, we propose `Program of Thoughts' (PoT), which uses language models (mainly Codex) to express the reasoning process as a program. The computation is relegated to an external computer, which executes the generated programs to derive the answer. We evaluate PoT on five math word problem datasets (GSM, AQuA, SVAMP, TabMWP, MultiArith) and three financial-QA datasets (FinQA, ConvFinQA, TATQA) for both few-shot and zero-shot setups. Under both few-shot and zero-shot settings, PoT can show an average performance gain over CoT by around 12% across all the evaluated datasets. By combining PoT with self-consistency decoding, we can achieve SoTA performance on all math problem datasets and near-SoTA performance on financial datasets. All of our data and code are released in Github https://github.com/wenhuchen/Program-of-Thoughts

Key facts. Cited by 1,057 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 39 of its 49 references that are in the map. First posted on arXiv 22 Nov 2022.

Builds on

39 of 49 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them17 Oct
  2. 2022Large Language Models are Zero-Shot Reasoners24 May
  3. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Models21 May
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  10. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  12. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
22 more
  1. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  2. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  3. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  4. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  5. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  6. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  7. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  8. 2022PAL: Program-aided Language Modelscited by 635
  9. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  10. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  11. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
  12. 2016Solving General Arithmetic Word Problemscited by 353
  13. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  14. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  15. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  16. 2022Binding Language Models in Symbolic Languagescited by 224
  17. 2022ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answeringcited by 194
  18. 2022Large Language Models are few(1)-shot Table Reasonerscited by 116
  19. 2022Compositional Semantic Parsing with Large Language Modelscited by 95
  20. 2017Mapping to Declarative Knowledge for Word Problem Solvingcited by 36
  21. 2022Answering Numerical Reasoning Questions in Table-Text Hybrid Contents with Graph-based Encoder and Tree-based Decodercited by 17
  22. 2022A Numerical Reasoning Question Answering System with Fine-grained Retriever and the Ensemble of Multiple Generators for FinQAcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 204; 2024: 304; 2025: 297; 2026: 243.

1,057 papers.

Sort:

By area:

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  3. 2023Teaching Large Language Models to Self-Debugcited by 963
  4. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  5. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  6. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  7. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  8. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  9. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  10. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  11. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  12. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
38 more
  1. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  2. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  3. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  4. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  5. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 428
  6. 2024Understanding the planning of LLM agents: A surveycited by 414
  7. 2023Augmented Language Models: a Surveycited by 398
  8. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
  9. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  10. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 379
  11. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 377
  12. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  13. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 348
  14. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  15. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
  16. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  17. 2023Tool Learning with Foundation Modelscited by 302
  18. 2023Large Language Models as Tool Makerscited by 293
  19. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  20. 2023Faithful Chain-of-Thought Reasoningcited by 285
  21. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 283
  22. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 281
  23. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 276
  24. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  25. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 270
  26. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 261
  27. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  28. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 256
  29. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  30. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
  31. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  32. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
  33. 2023Do Large Language Models Know What They Don't Know?cited by 238
  34. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  35. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  36. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 228
  37. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 222
  38. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221

50 most cited of 1,057 All 1,057 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Leveraging Training Data in Few-Shot Prompting for Numerical Reasoning
  2. 2025Structured Reasoning with Tree-of-Thoughts for Bengali Math Word Problems
  3. 2024Boosting of Thoughts: Trial-and-Error Problem Solving with Large Language Models
  4. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective
  5. 2026Evaluating Prompting and Execution-Based Methods for Deterministic Computation in LLMs
  6. 2023ThoughtSource: A central hub for large language model reasoning data
  7. 2022Teaching Small Language Models to Reason
  8. 2022Complexity-Based Prompting for Multi-Step Reasoning