# Decomposed Prompting Does Not Fix Knowledge Gaps, But Helps Models Say "I Don't Know"

arXiv: https://arxiv.org/abs/2602.04853 · HTML: https://mlsanity.com/paper/2602.04853/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 4 Feb 2026.

## Builds on (most cited first, 22 of 22)

- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [Decomposed Prompting: A Modular Approach for Solving Complex Tasks](https://mlsanity.com/paper/2210.02406/) (2022)
- [LM vs LM: Detecting Factual Errors via Cross Examination](https://mlsanity.com/paper/2305.13281/) (2023)
- [Break It Down: A Question Understanding Benchmark](https://mlsanity.com/paper/2001.11770/) (2020)
- [Mobile Edge Intelligence for Large Language Models: A Contemporary Survey](https://mlsanity.com/paper/2407.18921/) (2024)
- [Multi-hop Reading Comprehension through Question Decomposition and Rescoring](https://mlsanity.com/paper/1906.02916/) (2019)
- [SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistency](https://mlsanity.com/paper/2311.01740/) (2023)
- [Frames: A Corpus for Adding Memory to Goal-Oriented Dialogue Systems](https://mlsanity.com/paper/1704.00057/) (2017)
- [Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answering](https://mlsanity.com/paper/2210.01613/) (2022)
- [The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs](https://mlsanity.com/paper/2509.09677/) (2025)
- [Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QA](https://mlsanity.com/paper/1906.07132/) (2019)
- [Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Method](https://mlsanity.com/paper/2310.17918/) (2023)
- [Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answer](https://mlsanity.com/paper/2502.12964/) (2025)
- [MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documents](https://mlsanity.com/paper/2508.11133/) (2025)
- [SCORE: Systematic COnsistency and Robustness Evaluation for Large Language Models](https://mlsanity.com/paper/2503.00137/) (2025)
- [Weakly Supervised Text-to-SQL Parsing through Question Decomposition](https://mlsanity.com/paper/2112.06311/) (2021)
- [GenDec: A robust generative Question-decomposition method for Multi-hop reasoning](https://mlsanity.com/paper/2402.11166/) (2024)
- [Improving Consistency in Large Language Models through Chain of Guidance](https://mlsanity.com/paper/2502.15924/) (2025)
- [LLMs with Industrial Lens: Deciphering the Challenges and Prospects -- A Survey](https://mlsanity.com/paper/2402.14558/) (2024)

## Built on by (most cited first, 1 of 1)

- [Pay for the Fault, Not the Flow: Label-Free In-Flow Multi-Agent Workflow Optimization](https://mlsanity.com/paper/2610.01017/) (2026)

## Abstract (arXiv)

Large language models often struggle to recognize their knowledge limits in closed-book question answering, leading to confident hallucinations. While decomposed prompting is typically used to improve accuracy, we investigate its impact on reliability. We evaluate three task-equivalent prompting regimes: Direct, Assistive, and Incremental, across different model scales and multi-hop QA benchmarks. We find that although accuracy gains from decomposition diminish in frontier models, disagreements between prompting regimes remain highly indicative of potential errors. Because factual knowledge is typically stable while hallucinations are stochastic, cross-regime agreement provides a precise signal of internal uncertainty. We leverage this signal to implement a training-free abstention policy that requires no retrieval or fine-tuning. Our results show that disagreement-based abstention outperforms standard uncertainty baselines as an error detector, improving both F1 and AUROC across settings. This demonstrates that decomposition-based prompting can serve as a practical diagnostic probe for model reliability in closed-book QA.