mlsanityData to 14 Oct 2026

Assessing the Impact of Prompting Methods on ChatGPT's Mathematical Capabilities

arXiv 2312.15006 ↗First posted 22 Dec 2023cs.AI cs.CL cs.LGcited by 0

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

This study critically evaluates the efficacy of prompting methods in enhancing the mathematical reasoning capability of large language models (LLMs). The investigation uses three prescriptive prompting methods - simple, persona, and conversational prompting - known for their effectiveness in enhancing the linguistic tasks of LLMs. We conduct this analysis on OpenAI's LLM chatbot, ChatGPT-3.5, on extensive problem sets from the MATH, GSM8K, and MMLU datasets, encompassing a broad spectrum of mathematical challenges. A grading script adapted to each dataset is used to determine the effectiveness of these prompting interventions in enhancing the model's mathematical analysis power. Contrary to expectations, our empirical analysis reveals that none of the investigated methods consistently improves over ChatGPT-3.5's baseline performance, with some causing significant degradation. Our findings suggest that prompting strategies do not necessarily generalize to new domains, in this study failing to enhance mathematical performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 18 of its 27 references that are in the map. First posted on arXiv 22 Dec 2023.

Builds on

18 of 27 references.

References · 18

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2020Aligning AI With Shared Human Valuescited by 723
  7. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  8. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  9. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 462
  10. 2023Mathematical Capabilities of ChatGPTcited by 268
  11. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  12. 2022Domain Adaptation via Prompt Learningcited by 128
6 more
  1. 2022Long Time No See! Open-Domain Conversation with Long-Term Persona Memorycited by 108
  2. 2023Context-faithful Prompting for Large Language Modelscited by 80
  3. 2023ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Designcited by 66
  4. 2023Personas as a Way to Model Truthfulness in Language Modelscited by 47
  5. 2022Persona-Based Conversational AI: State of the Art and Challengescited by 6
  6. 2023Task Oriented Conversational Modelling With Subjective Knowledgecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Expanding Search Space with Diverse Prompting Agents: An Efficient Sampling Approach for LLM Mathematical Reasoning
  2. 2024Investigating Symbolic Capabilities of Large Language Models
  3. 2024Look Before You Leap: Problem Elaboration Prompting Improves Mathematical Reasoning in Large Language Models
  4. 2024Assessing the Emergent Symbolic Reasoning Abilities of Llama Large Language Models
  5. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers
  6. 2024MathChat: Benchmarking Mathematical Reasoning and Instruction Following in Multi-Turn Interactions
  7. 2025Probing Large Language Models in Reasoning and Translating Complex Linguistic Puzzles
  8. 2024Towards Generalist Prompting for Large Language Models by Mental Models