mlsanityData to 14 Oct 2026

Towards Context-Robust LLMs: A Gated Representation Fine-tuning Approach

arXiv 2502.14100 ↗First posted 19 Feb 2025cs.CL cs.IRcited by 6

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

Large Language Models (LLMs) enhanced with external contexts, such as through retrieval-augmented generation (RAG), often face challenges in handling imperfect evidence. They tend to over-rely on external knowledge, making them vulnerable to misleading and unhelpful contexts. To address this, we propose the concept of context-robust LLMs, which can effectively balance internal knowledge with external context, similar to human cognitive processes. Specifically, context-robust LLMs should rely on external context only when lacking internal knowledge, identify contradictions between internal and external knowledge, and disregard unhelpful contexts. To achieve this goal, we introduce Grft, a lightweight and plug-and-play gated representation fine-tuning approach. Grft consists of two key components: a gating mechanism to detect and filter problematic inputs, and low-rank representation adapters to adjust hidden representations. By training a lightweight intervention function with only 0.0004% of model size on fewer than 200 examples, Grft can effectively adapt LLMs towards context-robust behaviors.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 22 of its 31 references that are in the map. First posted on arXiv 19 Feb 2025.

Builds on

22 of 31 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Training31 May
  2. 2024RA-ISF: Learning to Answer and Understand from Retrieval Augmentation via Iterative Self-Feedback11 Mar
  3. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection17 Oct

Other references · 19

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  4. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  5. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  6. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 369
  7. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
  8. 2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 281
  9. 2024ReFT: Representation Finetuning for Language Modelscited by 197
  10. 2024The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)cited by 159
  11. 2023Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentationcited by 147
  12. 2024Blended RAG: Improving RAG (Retriever-Augmented Generation) Accuracy with Semantic Search and Hybrid Query-Based Retrieverscited by 84
7 more
  1. 2024CBR-RAG: Case-Based Reasoning for Retrieval Augmented Generation in LLMs for Legal Question Answeringcited by 72
  2. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 66
  3. 2024Pandora: Jailbreak GPTs by Retrieval Augmented Generation Poisoningcited by 64
  4. 2024Towards Understanding Jailbreak Attacks in LLMs: A Representation Space Analysiscited by 64
  5. 2024When Do LLMs Need Retrieval Augmentation? Mitigating LLMs' Overconfidence Helps Retrieval Augmentationcited by 59
  6. 2024Improving Retrieval for RAG based Question Answering Models on Financial Documentscited by 49
  7. 2024Towards Knowledge Checking in Retrieval-augmented Generation: A Representation Perspectivecited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Empowering GraphRAG with Knowledge Filtering and Integrationcited by 15
  2. 2025Beyond Text: Unveiling Privacy Vulnerabilities in Multi-modal Retrieval-Augmented Generationcited by 5
  3. 2026Attn-GS: Attention-Guided Context Compression for Efficient Personalized LLMscited by 1
  4. 2026Magnifying What Matters: Attention-Guided Adaptive Rendering for Visual Text Comprehensioncited by 1
  5. 2026PreFT: Prefill-only finetuning for efficient inferencecited by 0
  6. 2026Efficient LLM Adversarial Training via Low-Rank Defense and Circuit-Guided Surrogatescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Adaptive Distraction: Probing LLM Contextual Robustness with Automated Tree Search
  2. 2024Efficient LLM Context Distillation
  3. 2025LLM Inference Enhanced by External Knowledge: A Survey
  4. 2025Demystifying the Roles of LLM Layers in Retrieval, Knowledge, and Reasoning
  5. 2024Improving Large Language Model (LLM) fidelity through context-aware grounding: A systematic approach to reliability and veracity
  6. 2025RbFT: Robust Fine-tuning for Retrieval-Augmented Generation against Retrieval Defects
  7. 2024Large Language Models Can Self-Improve in Long-context Reasoning
  8. 2025A General Framework to Enhance Fine-tuning-based LLM Unlearning