# Enhancing LLM Reliability via Explicit Knowledge Boundary Modeling

arXiv: https://arxiv.org/abs/2503.02233 · HTML: https://mlsanity.com/paper/2503.02233/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 38 references that are in the map.
- First posted on arXiv 4 Mar 2025.

## Builds on (most cited first, 30 of 31)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset](https://mlsanity.com/paper/1909.05855/) (2019)
- [A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models](https://mlsanity.com/paper/2401.01313/) (2024)
- [We're Afraid Language Models Aren't Modeling Ambiguity](https://mlsanity.com/paper/2304.14399/) (2023)
- [Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthiness](https://mlsanity.com/paper/2308.16175/) (2023)
- [DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales](https://mlsanity.com/paper/2308.01320/) (2023)
- [Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty](https://mlsanity.com/paper/2401.06730/) (2024)
- [MultiWOZ 2.4: A Multi-Domain Task-Oriented Dialogue Dataset with Essential Annotation Corrections to Improve State Tracking Evaluation](https://mlsanity.com/paper/2104.00773/) (2021)
- [Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedback](https://mlsanity.com/paper/2403.18349/) (2024)
- [BiToD: A Bilingual Multi-Domain Dataset For Task-Oriented Dialogue Modeling](https://mlsanity.com/paper/2106.02787/) (2021)
- [A Survey of Uncertainty Estimation in LLMs: Theory Meets Practice](https://mlsanity.com/paper/2410.15326/) (2024)
- [When to Trust LLMs: Aligning Confidence with Response Quality](https://mlsanity.com/paper/2404.17287/) (2024)
- [Knowledge Boundary of Large Language Models: A Survey](https://mlsanity.com/paper/2412.12472/) (2024)
- [Reducing Tool Hallucination via Reliability Alignment](https://mlsanity.com/paper/2412.04141/) (2024)
- [Benchmarking Knowledge Boundary for Large Language Models: A Different Perspective on Model Evaluation](https://mlsanity.com/paper/2402.11493/) (2024)
- [Teaching Large Language Models to Express Knowledge Boundary from Their Own Signals](https://mlsanity.com/paper/2406.10881/) (2024)
- [UAlign: Leveraging Uncertainty Estimations for Factuality Alignment on Large Language Models](https://mlsanity.com/paper/2412.11803/) (2024)
- [An Automatic Graph Construction Framework based on Large Language Models for Recommendation](https://mlsanity.com/paper/2412.18241/) (2024)

## Built on by (most cited first, 13 of 13)

- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025)
- [TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning](https://mlsanity.com/paper/2509.25760/) (2025)
- [KnowRL: Exploring Knowledgeable Reinforcement Learning for Factuality](https://mlsanity.com/paper/2506.19807/) (2025)
- [Alignment for Efficient Tool Calling of Large Language Models](https://mlsanity.com/paper/2503.06708/) (2025)
- [ReliableMath: Benchmark of Reliable Mathematical Reasoning on Large Language Models](https://mlsanity.com/paper/2507.03133/) (2025)
- [DiSRouter: Distributed Self-Routing for LLM Selections](https://mlsanity.com/paper/2510.19208/) (2025)
- [Delusions of Large Language Models](https://mlsanity.com/paper/2503.06709/) (2025)
- [BAPO: Boundary-Aware Policy Optimization for Reliable Agentic Search](https://mlsanity.com/paper/2601.11037/) (2026)
- [Know More, Know Clearer: A Meta-Cognitive Framework for Knowledge Augmentation in Large Language Models](https://mlsanity.com/paper/2602.12996/) (2026)
- [Rethinking LLM Parametric Knowledge as Post-retrieval Confidence for Dynamic Retrieval and Reranking](https://mlsanity.com/paper/2509.06472/) (2025)
- [RAS: a Reliability Oriented Metric for Automatic Speech Recognition](https://mlsanity.com/paper/2604.24278/) (2026)
- [XFlow: An Executable Protocol Programming System for Reliable Multi-Agent Workflows](https://mlsanity.com/paper/2606.14790/) (2026)
- [To Answer or to Abstain: Mitigating Search-Agent Hallucinations via Abstention-Aware Reinforcement Learning](https://mlsanity.com/paper/2607.10738/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are prone to hallucination stemming from misaligned self-awareness, particularly when processing queries exceeding their knowledge boundaries. While existing mitigation strategies employ uncertainty estimation or query rejection mechanisms, they suffer from computational efficiency and sacrificed helpfulness. To address these issues, we propose the Explicit Knowledge Boundary Modeling (EKBM) framework, integrating fast and slow reasoning systems to harmonize reliability and usability. The framework first employs a fast-thinking model to generate confidence-labeled responses, enabling immediate utilization of high-confidence outputs, whereas uncertain predictions trigger a slow refinement model for accuracy improvement. To align model behavior with our proposed object, we propose a hybrid training pipeline, enhancing self-awareness without degrading task performance. Evaluations on dialogue state tracking tasks demonstrate that EKBM achieves superior model reliability over uncertainty-based baselines. Further analysis reveals that refinement substantially boosts accuracy while maintaining low computational overhead. The framework establishes a scalable paradigm for deploying reliable LLMs in error-sensitive applications, effectively balancing accuracy and practical utility.