# Large Language Models Struggle with Unreasonability in Math Problems

arXiv: https://arxiv.org/abs/2403.19346 · HTML: https://mlsanity.com/paper/2403.19346/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 10 of its 39 references that are in the map.
- First posted on arXiv 28 Mar 2024.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://mlsanity.com/paper/2309.12284/) (2023)
- [WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct](https://mlsanity.com/paper/2308.09583/) (2023)
- [Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models](https://mlsanity.com/paper/2305.04091/) (2023)
- [Complexity-Based Prompting for Multi-Step Reasoning](https://mlsanity.com/paper/2210.00720/) (2022)
- [LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Models](https://mlsanity.com/paper/2305.13711/) (2023)
- [OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoning](https://mlsanity.com/paper/2311.09724/) (2023)

## Built on by (most cited first, 18 of 18)

- [AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions](https://mlsanity.com/paper/2506.09038/) (2025)
- [Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklist](https://mlsanity.com/paper/2407.08733/) (2024)
- [A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models](https://mlsanity.com/paper/2509.03871/) (2025)
- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [Stop When Enough: Adaptive Early-Stopping for Chain-of-Thought Reasoning](https://mlsanity.com/paper/2510.10103/) (2025)
- [Knowing When to Abstain: Medical LLMs Under Clinical Uncertainty](https://mlsanity.com/paper/2601.12471/) (2026)
- [Benchmarking Multi-Step Legal Reasoning and Analyzing Chain-of-Thought Effects in Large Language Models](https://mlsanity.com/paper/2511.07979/) (2025)
- [Zero-resource Hallucination Detection for Text Generation via Graph-based Contextual Knowledge Triples Modeling](https://mlsanity.com/paper/2409.11283/) (2024)
- [ReliableMath: Benchmark of Reliable Mathematical Reasoning on Large Language Models](https://mlsanity.com/paper/2507.03133/) (2025)
- [VCSearch: Bridging the Gap Between Well-Defined and Ill-Defined Problems in Mathematical Reasoning](https://mlsanity.com/paper/2406.05055/) (2024)
- [Statistical Early Stopping for Reasoning Models](https://mlsanity.com/paper/2602.13935/) (2026)
- [Beyond Passive Critical Thinking: Fostering Proactive Questioning to Enhance Human-AI Collaboration](https://mlsanity.com/paper/2507.23407/) (2025)
- [The Yes-Man Syndrome: Benchmarking Abstention in Embodied Robotic Agents](https://mlsanity.com/paper/2605.20544/) (2026)
- [When Not to Answer: Evaluating Prompts on GPT Models for Effective Abstention in Unanswerable Math Word Problems](https://mlsanity.com/paper/2410.13029/) (2024)
- [TIPS: Topological Ill-Posedness Probing and Steering in Large Language Models](https://mlsanity.com/paper/2606.23590/) (2026)
- [FlowEdit: Information-Theoretic Control of LLM Reasoning Flows for Ill-posed Problems Involving Conflicts](https://mlsanity.com/paper/2607.20500/) (2026)
- [Ask, Condition or Abstain: Reinforcement Learning for Missing-Premise Reasoning](https://mlsanity.com/paper/2608.16554/) (2026)
- [Recognition-Refusal Misalignment in LLMs: Why Models Answer Structurally Unanswerable Questions](https://mlsanity.com/paper/2608.29109/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have shown remarkable success on a wide range of math and reasoning benchmarks. However, we observe that they often struggle when faced with unreasonable math problems. Instead of recognizing these issues, models frequently proceed as if the problem is well-posed, producing incorrect answers or falling into overthinking and verbose self-correction. To systematically investigate this overlooked vulnerability, we propose the Unreasonable Math Problems (UMP) benchmark, designed to evaluate LLMs' ability to detect and respond to unreasonable math problem statements. Based on extensive experiments covering 19 LLMs, we find that even state-of-the-art general models like GPT-4o achieve only a score of 0.6 on UMP. While reasoning models such as DeepSeek-R1 demonstrate a higher sensitivity to unreasonable inputs, this often comes at the cost of generating overly long and meaningless responses that fail to converge. We further explore prompting and fine-tuning methods, which offer partial improvements but also introduce trade-offs, shedding light on both the potential and limitations of LLMs in this challenging setting.