# Quantifying Document Impact in RAG-LLMs

arXiv: https://arxiv.org/abs/2601.05260 · HTML: https://mlsanity.com/paper/2601.05260/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Privacy and security of RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 25 of its 28 references that are in the map.
- First posted on arXiv 27 Oct 2025.

## Placed in

- Retrieval-augmented generation / [Privacy and security of RAG](https://mlsanity.com/rag/privacy-and-security-of-rag/)

## Builds on (most cited first, 25 of 25)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models](https://mlsanity.com/paper/2305.19187/) (2023)
- [PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models](https://mlsanity.com/paper/2402.07867/) (2024)
- [Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs](https://mlsanity.com/paper/2406.15927/) (2024)
- [Challenges and Applications of Large Language Models](https://mlsanity.com/paper/2307.10169/) (2023)
- [RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs](https://mlsanity.com/paper/2407.02485/) (2024)
- [Poisoning Retrieval Corpora by Injecting Adversarial Passages](https://mlsanity.com/paper/2310.19156/) (2023)
- [ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidence](https://mlsanity.com/paper/2404.10198/) (2024)
- [BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models](https://mlsanity.com/paper/2406.00083/) (2024)
- [RECALL: A Benchmark for LLMs Robustness against External Counterfactual Knowledge](https://mlsanity.com/paper/2311.08147/) (2023)
- [Unveiling the Implicit Toxicity in Large Language Models](https://mlsanity.com/paper/2311.17391/) (2023)
- [Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)](https://mlsanity.com/paper/2407.14937/) (2024)
- [Does RAG Introduce Unfairness in LLMs? Evaluating Fairness in Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2409.19804/) (2024)
- [RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Models](https://mlsanity.com/paper/2504.18041/) (2025)
- [Long Context RAG Performance of Large Language Models](https://mlsanity.com/paper/2411.03538/) (2024)
- [A Novel Approach to the Partial Information Decomposition](https://mlsanity.com/paper/1908.08642/) (2019)
- [CPA-RAG:Covert Poisoning Attacks on Retrieval-Augmented Generation in Large Language Models](https://mlsanity.com/paper/2505.19864/) (2025)
- [The RAG Paradox: A Black-Box Attack Exploiting Unintentional Vulnerabilities in Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2502.20995/) (2025)
- [Bias Amplification in RAG: Poisoning Knowledge Retrieval to Steer LLMs](https://mlsanity.com/paper/2506.11415/) (2025)
- [Mitigating Bias in RAG: Controlling the Embedder](https://mlsanity.com/paper/2502.17390/) (2025)

## Abstract (arXiv)

Retrieval Augmented Generation (RAG) enhances Large Language Models (LLMs) by connecting them to external knowledge, improving accuracy and reducing outdated information. However, this introduces challenges such as factual inconsistencies, source conflicts, bias propagation, and security vulnerabilities, which undermine the trustworthiness of RAG systems. A key gap in current RAG evaluation is the lack of a metric to quantify the contribution of individual retrieved documents to the final output. To address this, we introduce the Influence Score (IS), a novel metric based on Partial Information Decomposition that measures the impact of each retrieved document on the generated response. We validate IS through two experiments. First, a poison attack simulation across three datasets demonstrates that IS correctly identifies the malicious document as the most influential in 86% of cases. Second, an ablation study shows that a response generated using only the top-ranked documents by IS is consistently judged more similar to the original response than one generated from the remaining documents. These results confirm the efficacy of IS in isolating and quantifying document influence, offering a valuable tool for improving the transparency and reliability of RAG systems.