# When Not to Answer: Evaluating Prompts on GPT Models for Effective Abstention in Unanswerable Math Word Problems

arXiv: https://arxiv.org/abs/2410.13029 · HTML: https://mlsanity.com/paper/2410.13029/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 43 references that are in the map.
- First posted on arXiv 16 Oct 2024.

## Builds on (most cited first, 24 of 24)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Better Zero-Shot Reasoning with Role-Play Prompting](https://mlsanity.com/paper/2308.07702/) (2023)
- [A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4](https://mlsanity.com/paper/2310.12321/) (2023)
- [Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?](https://mlsanity.com/paper/2402.12483/) (2024)
- [Improving Large Language Model Fine-tuning for Solving Math Problems](https://mlsanity.com/paper/2310.10047/) (2023)
- [Efficient Prompting Methods for Large Language Models: A Survey](https://mlsanity.com/paper/2404.01077/) (2024)
- [ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipeline](https://mlsanity.com/paper/2404.02893/) (2024)
- [Benchmarking Hallucination in Large Language Models based on Unanswerable Math Word Problem](https://mlsanity.com/paper/2403.03558/) (2024)
- [When to Trust LLMs: Aligning Confidence with Response Quality](https://mlsanity.com/paper/2404.17287/) (2024)
- [How Susceptible are LLMs to Influence in Prompts?](https://mlsanity.com/paper/2408.11865/) (2024)
- [MathAttack: Attacking Large Language Models Towards Math Solving Ability](https://mlsanity.com/paper/2309.01686/) (2023)
- [An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP)](https://mlsanity.com/paper/2302.13814/) (2023)
- [Don't Just Say "I don't know"! Self-aligning Large Language Models for Responding to Unknown Questions with Explanations](https://mlsanity.com/paper/2402.15062/) (2024)
- [UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Models](https://mlsanity.com/paper/2310.10942/) (2023)
- [The Dark Side of ChatGPT: Legal and Ethical Challenges from Stochastic Parrots and Hallucination](https://mlsanity.com/paper/2304.14347/) (2023)
- [Can LLMs Solve longer Math Word Problems Better?](https://mlsanity.com/paper/2405.14804/) (2024)
- [Large Language Models Struggle with Unreasonability in Math Problems](https://mlsanity.com/paper/2403.19346/) (2024)
- [ChatGPT as a Math Questioner? Evaluating ChatGPT on Generating Pre-university Math Questions](https://mlsanity.com/paper/2312.01661/) (2023)
- [The Role of ChatGPT in Democratizing Data Science: An Exploration of AI-facilitated Data Analysis in Telematics](https://mlsanity.com/paper/2308.02045/) (2023)

## Built on by (most cited first, 2 of 2)

- [AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions](https://mlsanity.com/paper/2506.09038/) (2025)
- [Honesty over Accuracy: Trustworthy Language Models through Reinforced Hesitation](https://mlsanity.com/paper/2511.11500/) (2025)

## Abstract (arXiv)

Large language models (LLMs) are increasingly relied upon to solve complex mathematical word problems. However, being susceptible to hallucination, they may generate inaccurate results when presented with unanswerable questions, raising concerns about their potential harm. While GPT models are now widely used and trusted, the exploration of how they can effectively abstain from answering unanswerable math problems and the enhancement of their abstention capabilities has not been rigorously investigated. In this paper, we investigate whether GPTs can appropriately respond to unanswerable math word problems by applying prompts typically used in solvable mathematical scenarios. Our experiments utilize the Unanswerable Word Math Problem (UWMP) dataset, directly leveraging GPT model APIs. Evaluation metrics are introduced, which integrate three key factors: abstention, correctness and confidence. Our findings reveal critical gaps in GPT models and the hallucination it suffers from for unsolvable problems, highlighting the need for improved models capable of better managing uncertainty and complex reasoning in math word problem-solving contexts.