# Diversity Enhances an LLM's Performance in RAG and Long-context Task

arXiv: https://arxiv.org/abs/2502.09017 · HTML: https://mlsanity.com/paper/2502.09017/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 29 references that are in the map.
- First posted on arXiv 13 Feb 2025.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [Efficient Attentions for Long Document Summarization](https://mlsanity.com/paper/2104.02112/) (2021)
- [An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics](https://mlsanity.com/paper/2207.00939/) (2022)
- [Modular RAG: Transforming RAG Systems into LEGO-like Reconfigurable Frameworks](https://mlsanity.com/paper/2407.21059/) (2024)
- [GRETEL: Graph Contrastive Topic Enhanced Language Model for Long Document Extractive Summarization](https://mlsanity.com/paper/2208.09982/) (2022)

## Built on by (most cited first, 8 of 8)

- [DF-RAG: Query-Aware Diversity for Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.17212/) (2026)
- [Culinary Crossroads: A RAG Framework for Enhancing Diversity in Cross-Cultural Recipe Adaptation](https://mlsanity.com/paper/2507.21934/) (2025)
- [DIVERGE: Diversity-Enhanced RAG for Open-Ended Information Seeking](https://mlsanity.com/paper/2602.00238/) (2026)
- [How Fine-Grained Should a RAG Benchmark Be? A Hierarchical Framework for Synthetic Question Generation](https://mlsanity.com/paper/2606.12789/) (2026)
- [An Interactive Paradigm for Deep Research](https://mlsanity.com/paper/2605.24266/) (2026)
- [Search for Coverage: Learning Coverage-Aware Retrieval with Augmented Sub-Question Answerability](https://mlsanity.com/paper/2605.28522/) (2026)
- [The Laws of Context Allocation: Causal Measurement and Closed-Loop Orchestration in Generative Search](https://mlsanity.com/paper/2608.23252/) (2026)
- [Finding the Right Balance: Relevance and Diversity in LLM Retrieval](https://mlsanity.com/paper/2610.09412/) (2026)

## Abstract (arXiv)

The rapid advancements in large language models (LLMs) have highlighted the challenge of context window limitations, primarily due to the quadratic time complexity of the self-attention mechanism ((O(N2)), where (N) denotes the context window length). This constraint impacts tasks such as retrieval-augmented generation (RAG) in question answering (Q&A) and long context summarization. A common approach involves selecting content with the highest similarity to the query; however, this often leads to redundancy and the exclusion of diverse yet relevant information. Building on principles from Maximal Marginal Relevance (MMR) and Farthest Point Sampling (FPS), we integrate diversity into the content selection process. Our findings reveal that incorporating diversity substantially increases the recall of selecting relevant sentences or chunks before LLM-based Q&A and summarization. These results highlight the importance of maintaining diversity in future LLM applications to further improve summarization and Q&A outcomes.