# Better RAG using Relevant Information Gain

arXiv: https://arxiv.org/abs/2407.12101 · HTML: https://mlsanity.com/paper/2407.12101/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 11 of its 24 references that are in the map.
- First posted on arXiv 16 Jul 2024.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 11 of 11)

- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [Benchmarking Large Language Models in Retrieval-Augmented Generation](https://mlsanity.com/paper/2309.01431/) (2023)
- [The Power of Noise: Redefining Retrieval for RAG Systems](https://mlsanity.com/paper/2401.14887/) (2024)
- [A Theoretical Analysis of NDCG Type Ranking Measures](https://mlsanity.com/paper/1304.6480/) (2013)
- [ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback](https://mlsanity.com/paper/2404.00934/) (2024)
- [Dense Passage Retrieval: Is it Retrieving?](https://mlsanity.com/paper/2402.11035/) (2024)
- [Extractive Summarization of Legal Decisions using Multi-task Learning and Maximal Marginal Relevance](https://mlsanity.com/paper/2210.12437/) (2022)
- [Machine learning approach for text and document mining](https://mlsanity.com/paper/1406.1580/) (2014)
- [Diversity Aware Relevance Learning for Argument Search](https://mlsanity.com/paper/2011.02177/) (2020)
- [Content Based Image Retrieval System using Feature Classification with Modified KNN Algorithm](https://mlsanity.com/paper/1307.4717/) (2013)

## Built on by (most cited first, 8 of 8)

- [RAG+: Enhancing Retrieval-Augmented Generation with Application-Aware Reasoning](https://mlsanity.com/paper/2506.11555/) (2025)
- [MUSS: Multilevel Subset Selection for Relevance and Diversity](https://mlsanity.com/paper/2503.11126/) (2025)
- [DIP: Dynamic In-Context Planner For Diffusion Language Models](https://mlsanity.com/paper/2601.03199/) (2026)
- [SIFT: Selective-Index For Fast Compute of RAG Prefill by Exploiting Attention Invariance](https://mlsanity.com/paper/2606.09441/) (2026)
- [Less can be More: Relieving RAG Bottlenecks via Evidence Frontloading and Pressure-Adaptive Budgeting](https://mlsanity.com/paper/2608.25115/) (2026)
- [Isometry pursuit](https://mlsanity.com/paper/2411.18502/) (2024)
- [CoRAG: Collaborative Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.01883/) (2025)
- [Finding the Right Balance: Relevance and Diversity in LLM Retrieval](https://mlsanity.com/paper/2610.09412/) (2026)

## Abstract (arXiv)

A common way to extend the memory of large language models (LLMs) is by retrieval augmented generation (RAG), which inserts text retrieved from a larger memory into an LLM's context window. However, the context window is typically limited to several thousand tokens, which limits the number of retrieved passages that can inform a model's response. For this reason, it's important to avoid occupying context window space with redundant information by ensuring a degree of diversity among retrieved passages. At the same time, the information should also be relevant to the current task. Most prior methods that encourage diversity among retrieved results, such as Maximal Marginal Relevance (MMR), do so by incorporating an objective that explicitly trades off diversity and relevance. We propose a novel simple optimization metric based on relevant information gain, a probabilistic measure of the total information relevant to a query for a set of retrieved results. By optimizing this metric, diversity organically emerges from our system. When used as a drop-in replacement for the retrieval component of a RAG system, this method yields state-of-the-art performance on question answering tasks from the Retrieval Augmented Generation Benchmark (RGB), outperforming existing metrics that directly optimize for relevance and diversity.