mlsanityData to 7 Oct 2026

Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedback

arXiv 2403.18349 ↗First posted 27 Mar 2024cs.CLcited by 64

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Large Language Models (LLMs) often generate erroneous outputs, known as hallucinations, due to their limitations in discerning questions beyond their knowledge scope. While addressing hallucination has been a focal point in research, previous efforts primarily concentrate on enhancing correctness without giving due consideration to the significance of rejection mechanisms. In this paper, we conduct a comprehensive examination of the role of rejection, introducing the notion of model reliability along with corresponding metrics. These metrics measure the model's ability to provide accurate responses while adeptly rejecting questions exceeding its knowledge boundaries, thereby minimizing hallucinations. To improve the inherent reliability of LLMs, we present a novel alignment framework called Reinforcement Learning from Knowledge Feedback (RLKF). RLKF leverages knowledge feedback to dynamically determine the model's knowledge boundary and trains a reliable reward model to encourage the refusal of out-of-knowledge questions. Experimental results on mathematical questions affirm the substantial efficacy of RLKF in significantly enhancing LLM reliability.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 41 of its 51 references that are in the map. First posted on arXiv 27 Mar 2024.

Builds on

41 of 51 references.

References · 41

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  12. 2020Learning to summarize from human feedbackcited by 2,562
29 more
  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  3. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  4. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  5. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  6. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  7. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  9. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  10. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  11. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  12. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  13. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  14. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  15. 2023How Language Model Hallucinations Can Snowballcited by 310
  16. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  17. 2023Challenges and Applications of Large Language Modelscited by 251
  18. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 179
  19. 2023Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Frameworkcited by 139
  20. 2021A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generationcited by 137
  21. 2023Hallucinations in Large Multilingual Translation Modelscited by 126
  22. 2023Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigationcited by 123
  23. 2021Truthful AI: Developing and governing AI that does not liecited by 109
  24. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scalescited by 107
  25. 2023Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Taskscited by 84
  26. 2023Alignment for Honestycited by 72
  27. 2024Calibrating Large Language Models with Sample Consistencycited by 72
  28. 2023Improving the Reliability of Large Language Models by Leveraging Uncertainty-Aware In-Context Learningcited by 29
  29. 2024On the Essence and Prospect: An Investigation of Alignment Approaches for Big Modelscited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 28; 2026: 16.

64 papers.

Sort:

By area:

  1. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  2. 2024Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?cited by 85
  3. 2024A Survey of Uncertainty Estimation in LLMs: Theory Meets Practicecited by 54
  4. 2025Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Modelscited by 48
  5. 2024Reducing Tool Hallucination via Reliability Alignmentcited by 37
  6. 2025Refusal Direction is Universal Across Safety-Aligned Languagescited by 37
  7. 2024Knowledge Boundary of Large Language Models: A Surveycited by 36
  8. 2025MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Modelscited by 33
  9. 2025Security Concerns for Large Language Models: A Surveycited by 26
  10. 2025Mask-DPO: Generalizable Fine-grained Factuality Alignment of LLMscited by 22
  11. 2024A Survey on the Honesty of Large Language Modelscited by 21
  12. 2024Perception of Knowledge Boundary for Large Language Models through Semi-open-ended Question Answeringcited by 20
38 more
  1. 2025TruthRL: Incentivizing Truthful LLMs via Reinforcement Learningcited by 20
  2. 2024ATM: Adversarial Tuning Multi-agent System Makes a Robust Retrieval-Augmented Generatorcited by 19
  3. 2025Alignment for Efficient Tool Calling of Large Language Modelscited by 17
  4. 2025KnowRL: Exploring Knowledgeable Reinforcement Learning for Factualitycited by 17
  5. 2024MobA: Multifaceted Memory-Enhanced Adaptive Planning for Efficient Mobile Task Automationcited by 14
  6. 2024UAlign: Leveraging Uncertainty Estimations for Factuality Alignment on Large Language Modelscited by 13
  7. 2025Enhancing LLM Reliability via Explicit Knowledge Boundary Modelingcited by 13
  8. 2025Language Model Uncertainty Quantification with Attention Chaincited by 13
  9. 2025Divide-Then-Align: Honest Alignment based on the Knowledge Boundary of RAGcited by 13
  10. 2024Wait, that's not an option: LLMs Robustness with Incorrect Multiple-Choice Optionscited by 11
  11. 2025GRAIT: Gradient-Driven Refusal-Aware Instruction Tuning for Effective Hallucination Mitigationcited by 11
  12. 2024Utilize the Flow before Stepping into the Same River Twice: Certainty Represented Knowledge Flow for Refusal-Aware Instruction Tuningcited by 10
  13. 2024On-Policy Self-Alignment with Fine-grained Knowledge Feedback for Hallucination Mitigationcited by 8
  14. 2024Unlocking the Power of LLM Uncertainty for Active In-Context Example Selectioncited by 8
  15. 2025BARREL: Boundary-Aware Reasoning for Factual and Reliable LRMscited by 8
  16. 2024When to Speak, When to Abstain: Contrastive Decoding with Abstentioncited by 7
  17. 2025Do We Know What LLMs Don't Know? A Study of Consistency in Knowledge Probingcited by 7
  18. 2024Regression for the Mean: Auto-Evaluation and Inference with Few Labels through Post-hoc Regressioncited by 6
  19. 2025SATER: A Self-Aware and Token-Efficient Approach to Routing and Cascadingcited by 6
  20. 2024Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captionscited by 5
  21. 2025Answer, Refuse, or Guess? Investigating Risk-Aware Decision Making in Language Modelscited by 5
  22. 2025Can LLMs Refuse Questions They Do Not Know? Measuring Knowledge-Aware Refusal in Factual Taskscited by 5
  23. 2025DiSRouter: Distributed Self-Routing for LLM Selectionscited by 5
  24. 2025Delusions of Large Language Modelscited by 4
  25. 2024Tell Me What You Don't Know: Enhancing Refusal Capabilities of Role-Playing Agents via Representation Space Analysis and Editingcited by 3
  26. 2025Balancing Truthfulness and Informativeness with Uncertainty-Aware Instruction Fine-Tuningcited by 3
  27. 2025Towards Reliable Large Audio Language Modelcited by 3
  28. 2025NeuSym-RAG: Hybrid Neural Symbolic Retrieval with Multiview Structuring for PDF Question Answeringcited by 3
  29. 2025MASH: Modeling Abstention via Selective Help-Seekingcited by 3
  30. 2026BAPO: Boundary-Aware Policy Optimization for Reliable Agentic Searchcited by 3
  31. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  32. 2024Probing LLM Hallucination from Within: Perturbation-Driven Approach via Internal Knowledgecited by 2
  33. 2026LLM-Confidence Reranker: A Training-Free Approach for Enhancing Retrieval-Augmented Generation Systemscited by 2
  34. 2024Reducing the Scope of Language Modelscited by 1
  35. 2024Drawing the Line: Enhancing Trustworthiness of MLLMs Through the Power of Refusalcited by 1
  36. 2025Dist2ill: Distributional Distillation for One-Pass Uncertainty Estimation in Large Language Modelscited by 1
  37. 2026Pardon? Evaluating Conversational Repair in Large Audio-Language Modelscited by 1
  38. 2026Abstention as an Action Can Kill Both the Reward Gradient and the KL Anchor: Collapse Law and Repair for Error-Penalized Reinforcement Learningcited by 1

50 most cited of 64 All 64 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Thinking, Faithful and Stable: Mitigating Hallucinations in LLMs
  2. 2024Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinations
  3. 2025The Hallucination Tax of Reinforcement Finetuning
  4. 2026Beyond Reasoning: Reinforcement Learning Unlocks Parametric Knowledge in LLMs
  5. 2026Reinforcement Inference: Leveraging Uncertainty for Self-Correcting Language Model Reasoning
  6. 2025High Accuracy, Less Talk (HALT): Reliable LLMs through Capability-Aligned Finetuning
  7. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback
  8. 2025Post-Training Large Language Models via Reinforcement Learning from Self-Feedback