mlsanityData to 7 Oct 2026

Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification

arXiv 2308.07921 ↗First posted 15 Aug 2023cs.CL cs.AI cs.CVcited by 166

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Recent progress in large language models (LLMs) like GPT-4 and PaLM-2 has brought significant advancements in addressing math reasoning problems. In particular, OpenAI's latest version of GPT-4, known as GPT-4 Code Interpreter, shows remarkable performance on challenging math datasets. In this paper, we explore the effect of code on enhancing LLMs' reasoning capability by introducing different constraints on the textitCode Usage Frequency of GPT-4 Code Interpreter. We found that its success can be largely attributed to its powerful skills in generating and executing code, evaluating the output of code execution, and rectifying its solution when receiving unreasonable outputs. Based on this insight, we propose a novel and effective prompting method, explicit ulinecode-based ulineself-ulineverification~(CSV), to further boost the mathematical reasoning potential of GPT-4 Code Interpreter. This method employs a zero-shot prompt on GPT-4 Code Interpreter to encourage it to use code to self-verify its answers. In instances where the verification state registers as ``False'', the model shall automatically amend its solution, analogous to our approach of rectifying errors during a mathematics examination. Furthermore, we recognize that the states of the verification result indicate the confidence of a solution, which can improve the effectiveness of majority voting. With GPT-4 Code Interpreter and CSV, we achieve an impressive zero-shot accuracy on MATH dataset (53.9% → 84.3%).

Key facts. Cited by 166 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 29 references that are in the map. First posted on arXiv 15 Aug 2023.

Builds on

25 of 29 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning29 Sep
  2. 2022Solving Quantitative Reasoning Problems with Language Models29 Jun

Other references · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2023Let's Verify Step by Stepcited by 3,598
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  11. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
11 more
  1. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  2. 2023PaLM 2 Technical Reportcited by 1,154
  3. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  4. 2022Galactica: A Large Language Model for Sciencecited by 736
  5. 2022PAL: Program-aided Language Modelscited by 635
  6. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  7. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  8. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 231
  9. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 189
  10. 2023Progressive-Hint Prompting Improves Reasoning in Large Language Modelscited by 147
  11. 2023Automatic Model Selection with Large Language Models for Reasoningcited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 26; 2024: 89; 2025: 39; 2026: 12.

166 papers.

Sort:

By area:

  1. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  2. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  3. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  4. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  5. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  6. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 283
  7. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 276
  8. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  9. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 257
  10. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  11. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 222
  12. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158
38 more
  1. 2023MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoningcited by 155
  2. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 155
  3. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 154
  4. 2025Reasoning Models Know When They're Right: Probing Hidden States for Self-Verificationcited by 154
  5. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143
  6. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 140
  7. 2024Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solverscited by 137
  8. 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospectscited by 130
  9. 2024Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-stepcited by 123
  10. 2023Chain of Code: Reasoning with a Language Model-Augmented Code Emulatorcited by 120
  11. 2024InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoningcited by 117
  12. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Enginecited by 111
  13. 2023Large Language Models as Analogical Reasonerscited by 107
  14. 2024GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinementscited by 101
  15. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 92
  16. 2024Small Language Models Need Strong Verifiers to Self-Correct Reasoningcited by 85
  17. 2023OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoningcited by 84
  18. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipelinecited by 81
  19. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
  20. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  21. 2024MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMscited by 72
  22. 2024Feedback Loops With Language Models Drive In-Context Reward Hackingcited by 70
  23. 2024OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AIcited by 70
  24. 2024SciAgent: Tool-augmented Language Models for Scientific Reasoningcited by 69
  25. 2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 68
  26. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 67
  27. 2024StateFlow: Enhancing LLM Task-Solving through State-Driven Workflowscited by 65
  28. 2025AgentRxiv: Towards Collaborative Autonomous Researchcited by 64
  29. 2024Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learningcited by 63
  30. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  31. 2024Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoningcited by 58
  32. 2024Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluationcited by 56
  33. 2024EmoBench: Evaluating the Emotional Intelligence of Large Language Modelscited by 53
  34. 2023Improving Large Language Model Fine-tuning for Solving Math Problemscited by 48
  35. 2024Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correctioncited by 45
  36. 2024CHAMP: A Competition-level Dataset for Fine-Grained Analyses of LLMs' Mathematical Reasoning Capabilitiescited by 43
  37. 2024MACM: Utilizing a Multi-Agent System for Condition Mining in Solving Complex Mathematical Problemscited by 41
  38. 2024Think Twice Before Trusting: Self-Detection for Large Language Models through Comprehensive Answer Reflectioncited by 40

50 most cited of 166 All 166 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023VerityMath: Advancing Mathematical Reasoning by Self-Verification Through Unit Consistency
  2. 2024ReasonAgain: Using Extractable Symbolic Programs to Evaluate Mathematical Reasoning
  3. 2024JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models
  4. 2023Assessing the Impact of Prompting Methods on ChatGPT's Mathematical Capabilities
  5. 2023Reasoning in Large Language Models Through Symbolic Math Word Problems
  6. 2023MathPrompter: Mathematical Reasoning using Large Language Models
  7. 2025Evaluation of LLMs for mathematical problem solving
  8. 2024Gap-Filling Prompting Enhances Code-Assisted Mathematical Reasoning