mlsanityData to 7 Oct 2026

GRAIT: Gradient-Driven Refusal-Aware Instruction Tuning for Effective Hallucination Mitigation

arXiv 2502.05911 ↗First posted 9 Feb 2025cs.CLcited by 11

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Refusal-Aware Instruction Tuning (RAIT) aims to enhance Large Language Models (LLMs) by improving their ability to refuse responses to questions beyond their knowledge, thereby reducing hallucinations and improving reliability. Effective RAIT must address two key challenges: firstly, effectively reject unknown questions to minimize hallucinations; secondly, avoid over-refusal to ensure questions that can be correctly answered are not rejected, thereby maintain the helpfulness of LLM outputs. In this paper, we address the two challenges by deriving insightful observations from the gradient-based perspective, and proposing the Gradient-driven Refusal Aware Instruction Tuning Framework GRAIT: (1) employs gradient-driven sample selection to effectively minimize hallucinations and (2) introduces an adaptive weighting mechanism during fine-tuning to reduce the risk of over-refusal, achieving the balance between accurate refusals and maintaining useful responses. Experimental evaluations on open-ended and multiple-choice question answering tasks demonstrate that GRAIT significantly outperforms existing RAIT methods in the overall performance. The source code and data will be available at https://github.com/opendatalab/GRAIT .

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 35 of its 55 references that are in the map. First posted on arXiv 9 Feb 2025.

Builds on

35 of 55 references.

References · 35

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  11. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  12. 2020Dataset Condensation with Gradient Matchingcited by 510
23 more
  1. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  2. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
  3. 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 269
  4. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222
  5. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 176
  6. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 156
  7. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  8. 2024Unfamiliar Finetuning Examples Control How Language Models Hallucinatecited by 85
  9. 2023Successor Heads: Recurring, Interpretable Attention Heads In The Wildcited by 77
  10. 2024Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedbackcited by 64
  11. 2023The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensivenesscited by 63
  12. 2024Identifying Semantic Induction Heads to Understand In-Context Learningcited by 46
  13. 2024Learning or Self-aligning? Rethinking Instruction Fine-tuningcited by 43
  14. 2024SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Modelscited by 43
  15. 2020Understanding Why Neural Networks Generalize Well Through GSNR of Parameterscited by 40
  16. 2024Teaching Large Language Models to Express Knowledge Boundary from Their Own Signalscited by 32
  17. 2024TC-RAG:Turing-Complete RAG's Case study on Medical LLM Systemscited by 28
  18. 2023HyKGE: A Hypothesis Knowledge Graph Enhanced Framework for Accurate and Reliable Medical LLMs Responsescited by 25
  19. 2024A Survey on the Honesty of Large Language Modelscited by 21
  20. 2024RAT: Adversarial Attacks on Deep Reinforcement Agents for Targeted Behaviorscited by 16
  21. 2024Parenting: Optimizing Knowledge Selection of Retrieval-Augmented Language Models with Parameter Decoupling and Tailored Tuningcited by 14
  22. 2024Uncertainty Aware Learning for Language Model Alignmentcited by 13
  23. 2024Efficient Model-agnostic Alignment via Bayesian Persuasioncited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 7.

11 papers.

  1. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 45
  2. 2025Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMscited by 29
  3. 2025AdaptFlow: Adaptive Workflow Optimization via Meta-Learningcited by 4
  4. 2026Hallucination is a Consequence of Space-Optimality: A Rate-Distortion Theorem for Membership Testingcited by 2
  5. 2026Divide and Cooperate: Role-Decomposed Multi-Agent LLM Training with Cross-Agent Learning Signalscited by 2
  6. 2026KbSD: Knowledge Boundary aware Self-Distillation for Behavioral Calibration in Agentic Searchcited by 2
  7. 2025Understanding New-Knowledge-Induced Factual Hallucinations in LLMs: Analysis and Interpretationcited by 0
  8. 2026Retrieval Augmented Generation Framework for the Nepali Legal Domain Question Answeringcited by 0
  9. 2026What Makes Agent Memory Useful for Reliable Unanswerable Question Handling?cited by 0
  10. 2026From Reweighting to Rewriting: Unlocking the Intervention Effects of Influential Samples in Training Data Attributioncited by 0
  11. 2026Learning Dynamics of Continual Learning: A Unified View of Data Attribution, Forgetting, and Plasticity Losscited by 0