mlsanityData to 14 Oct 2026

Divide-Then-Align: Honest Alignment based on the Knowledge Boundary of RAG

arXiv 2505.20871 ↗First posted 27 May 2025cs.CLcited by 14

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

also in 1 more
Abstract (arXiv)

Large language models (LLMs) augmented with retrieval systems have significantly advanced natural language processing tasks by integrating external knowledge sources, enabling more accurate and contextually rich responses. To improve the robustness of such systems against noisy retrievals, Retrieval-Augmented Fine-Tuning (RAFT) has emerged as a widely adopted method. However, RAFT conditions models to generate answers even in the absence of reliable knowledge. This behavior undermines their reliability in high-stakes domains, where acknowledging uncertainty is critical. To address this issue, we propose Divide-Then-Align (DTA), a post-training approach designed to endow RAG systems with the ability to respond with "I don't know" when the query is out of the knowledge boundary of both the retrieved passages and the model's internal knowledge. DTA divides data samples into four knowledge quadrants and constructs tailored preference data for each quadrant, resulting in a curated dataset for Direct Preference Optimization (DPO). Experimental results on three benchmark datasets demonstrate that DTA effectively balances accuracy with appropriate abstention, enhancing the reliability and trustworthiness of retrieval-augmented systems.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 36 of its 65 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

36 of 65 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Parameters vs. Context: Fine-Grained Control of Knowledge Reliance in Language Models20 Mar
  2. 2024Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Training31 May
  3. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity21 Mar
  4. 2024Enhancing LLM Factual Accuracy with RAG to Counter Hallucinations: A Case Study on Domain-Specific Queries in Private Knowledge-Bases15 Mar
  5. 2024ChatQA: Surpassing GPT-4 on Conversational QA and RAG18 Jan
  6. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection17 Oct

Other references · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2017On Calibration of Modern Neural Networkscited by 4,960
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  8. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  11. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  12. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
18 more
  1. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  2. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  3. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  4. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  5. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  6. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 369
  7. 2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 281
  8. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 262
  9. 2023A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generationcited by 206
  10. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 180
  11. 2024Faithful Logical Reasoning via Symbolic Chain-of-Thoughtcited by 100
  12. 2024Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedbackcited by 66
  13. 2024When Do LLMs Need Retrieval Augmentation? Mitigating LLMs' Overconfidence Helps Retrieval Augmentationcited by 59
  14. 2024Context-DPO: Aligning Language Models for Context-Faithfulnesscited by 50
  15. 2024Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refusecited by 34
  16. 2025Towards Fully Exploiting LLM Internal States to Enhance Knowledge Boundary Perceptioncited by 27
  17. 2025Innate Reasoning is Not Enough: In-Context Learning Enhances Reasoning Large Language Models with Less Overthinkingcited by 20
  18. 2024Is Factuality Enhancement a Free Lunch For LLMs? Better Factuality Can Lead to Worse Context-Faithfulnesscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 11.

14 papers.

Sort:

By area:

  1. 2026NOVA: NOise-aware Verbal Confidence CAlibration for Robust Large Language Models in RAG Systemscited by 6
  2. 2025HetaRAG: Hybrid Deep Retrieval-Augmented Generation across Heterogeneous Data Storescited by 3
  3. 2025Redefining Retrieval Evaluation in the Era of LLMscited by 3
  4. 2026BAPO: Boundary-Aware Policy Optimization for Reliable Agentic Searchcited by 3
  5. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  6. 2026Predict the Retrieval! Test time adaptation for Retrieval Augmented Generationcited by 2
  7. 2026FAITH: Factuality Alignment through Integrating Trustworthiness and Honestnesscited by 1
  8. 2026ERA: Evidence-based Reliability Alignment for Honest Retrieval-Augmented Generationcited by 1
  9. 2026GAPD: Gold-Action Policy Distillation for Agentic Reinforcement Learning in Knowledge Base Question Answeringcited by 1
  10. 2026Twin Worlds: Equivariance-Based Abstention for Evidence-Grounded Reasoningcited by 1
  11. 2025Rethinking LLM Parametric Knowledge as Post-retrieval Confidence for Dynamic Retrieval and Rerankingcited by 0
  12. 2026Training-Free Refusal of MCP Exploits via Retrieval-Augmented Generationcited by 0
2 more
  1. 2026Sieve and Sage: Efficient Distraction Filtering for Reliable RALM Abstentioncited by 0
  2. 2026Regime Boundary Alignment for Evidence-Gated Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation
  2. 2025RbFT: Robust Fine-tuning for Retrieval-Augmented Generation against Retrieval Defects
  3. 2025After Retrieval, Before Generation: Enhancing the Trustworthiness of Large Language Models in Retrieval-Augmented Generation
  4. 2025RoseRAG: Robust Retrieval-augmented Generation with Small-scale LLMs via Margin-aware Preference Optimization
  5. 2025Multi-hop Reasoning via Early Knowledge Alignment
  6. 2026URAG: A Benchmark for Uncertainty Quantification in Retrieval-Augmented Large Language Models
  7. 2026Trust or Abstain? A Self-Aware RAG Approach
  8. 2025Accommodate Knowledge Conflicts in Retrieval-augmented LLMs: Towards Robust Response Generation in the Wild