mlsanityData to 14 Oct 2026

Design of Chain-of-Thought in Math Problem Solving

arXiv 2309.11054 ↗First posted 20 Sep 2023cs.CL cs.AI cs.LGcited by 12

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Chain-of-Thought (CoT) plays a crucial role in reasoning for math problem solving. We conduct a comprehensive examination of methods for designing CoT, comparing conventional natural language CoT with various program CoTs, including the self-describing program, the comment-describing program, and the non-describing program. Furthermore, we investigate the impact of programming language on program CoTs, comparing Python and Wolfram Language. Through extensive experiments on GSM8K, MATHQA, and SVAMP, we find that program CoTs often have superior effectiveness in math problem solving. Notably, the best performing combination with 30B parameters beats GPT-3.5-turbo by a significant margin. The results show that self-describing program offers greater diversity and thus can generally achieve higher performance. We also find that Python is a better choice of language than Wolfram for program CoTs. The experimental results provide a valuable guideline for future CoT designs that take into account both programming language and coding style for further advancements. Our datasets and code are publicly available.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 38 of its 41 references that are in the map. First posted on arXiv 20 Sep 2023.

Builds on

38 of 41 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks22 Nov
  2. 2022Complexity-Based Prompting for Multi-Step Reasoning3 Oct
  3. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Models21 May
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  8. 2023Let's Verify Step by Stepcited by 3,789
  9. 2022Emergent Abilities of Large Language Modelscited by 2,512
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  11. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  12. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
21 more
  1. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  2. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  3. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  4. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  5. 2022Galactica: A Large Language Model for Sciencecited by 744
  6. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  7. 2022PAL: Program-aided Language Modelscited by 652
  8. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  9. 2021MetaICL: Learning to Learn In Contextcited by 454
  10. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 256
  11. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  12. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 177
  13. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificationcited by 166
  14. 2017Controlling Linguistic Style Aspects in Neural Language Generationcited by 149
  15. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  16. 2023Solving Math Word Problems by Combining Language Models With Symbolic Solverscited by 124
  17. 2021A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Levelcited by 96
  18. 2022Structured Prompting: Scaling In-Context Learning to 1,000 Examplescited by 81
  19. 2023Meta-in-context learning in large language modelscited by 59
  20. 2022Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extractioncited by 44
  21. 2021Investigating Math Word Problems using Pretrained Multilingual Language Modelscited by 21

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 6; 2025: 5.

12 papers.

  1. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 239
  2. 2024Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learningcited by 65
  3. 2025ChemAgent: Self-updating Library in Large Language Models Improves Chemical Reasoningcited by 41
  4. 2023Towards A Unified View of Answer Calibration for Multi-Step Reasoningcited by 8
  5. 2025SemCoT: Accelerating Chain-of-Thought Reasoning through Semantically-Aligned Implicit Tokenscited by 8
  6. 2024FLARE: Faithful Logic-Aided Reasoning and Explorationcited by 7
  7. 2024KARPA: A Training-free Method of Adapting Knowledge Graph as References for Large Language Model's Reasoning Path Aggregationcited by 7
  8. 2025Unlocking the Essence of Beauty: Advanced Aesthetic Reasoning with Relative-Absolute Policy Optimizationcited by 7
  9. 2024DCR: Divide-and-Conquer Reasoning for Multi-choice Question Answering with LLMscited by 5
  10. 2024SAAS: Solving Ability Amplification Strategy for Enhanced Mathematical Reasoning in Large Language Modelscited by 0
  11. 2025Reasoning with Preference Constraints: A Benchmark for Language Models in Many-to-One Matching Marketscited by 0
  12. 2025Parrot: A Training Pipeline Enhances Both Program CoT and Natural Language CoT for Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning
  2. 2025Chain-of-Thought Tokens are Computer Program Variables
  3. 2026The Potential of CoT for Reasoning: A Closer Look at Trace Dynamics
  4. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective
  5. 2024Markov Chain of Thought for Efficient Mathematical Reasoning
  6. 2025Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning
  7. 2024CoMAT: Chain of Mathematically Annotated Thought Improves Mathematical Reasoning
  8. 2025Structured Reasoning with Tree-of-Thoughts for Bengali Math Word Problems