# On the Role of Long-tail Knowledge in Retrieval Augmented Large Language Models

arXiv: https://arxiv.org/abs/2406.16367 · HTML: https://mlsanity.com/paper/2406.16367/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 21 of its 31 references that are in the map.
- First posted on arXiv 24 Jun 2024.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 21 of 21)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://mlsanity.com/paper/2212.10509/) (2022)
- [REPLUG: Retrieval-Augmented Black-Box Language Models](https://mlsanity.com/paper/2301.12652/) (2023)
- [Large Language Models Struggle to Learn Long-Tail Knowledge](https://mlsanity.com/paper/2211.08411/) (2022)
- [Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy](https://mlsanity.com/paper/2305.15294/) (2023)
- [Improving Calibration for Long-Tailed Recognition](https://mlsanity.com/paper/2104.00466/) (2021)
- [RA-DIT: Retrieval-Augmented Dual Instruction Tuning](https://mlsanity.com/paper/2310.01352/) (2023)
- [Learning to Filter Context for Retrieval-Augmented Generation](https://mlsanity.com/paper/2311.08377/) (2023)
- [UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation](https://mlsanity.com/paper/2303.08518/) (2023)
- [Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study](https://mlsanity.com/paper/2304.06762/) (2023)
- [Towards Calibrated Model for Long-Tailed Visual Recognition from Prior Perspective](https://mlsanity.com/paper/2111.03874/) (2021)
- [FEND: A Future Enhanced Distribution-Aware Contrastive Learning Framework for Long-tail Trajectory Prediction](https://mlsanity.com/paper/2303.16574/) (2023)
- [Balanced Product of Calibrated Experts for Long-Tailed Recognition](https://mlsanity.com/paper/2206.05260/) (2022)
- [Distantly-Supervised Long-Tailed Relation Extraction Using Constraint Graphs](https://mlsanity.com/paper/2105.11225/) (2021)
- [GradTail: Learning Long-Tailed Data Using Gradient-based Sample Weighting](https://mlsanity.com/paper/2201.05938/) (2022)

## Built on by (most cited first, 10 of 10)

- [Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.14891/) (2025)
- [UncertaintyRAG: Span-Level Uncertainty Enhanced Long-Context Modeling for Retrieval-Augmented Generation](https://mlsanity.com/paper/2410.02719/) (2024)
- [Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Models](https://mlsanity.com/paper/2507.02870/) (2025)
- [Parse Trees Guided LLM Prompt Compression](https://mlsanity.com/paper/2409.15395/) (2024)
- [LPFQA: A Long-Tail Professional Forum-based Benchmark for LLM Evaluation](https://mlsanity.com/paper/2511.06346/) (2025)
- [QCG-Rerank: Chunks Graph Rerank with Query Expansion in Retrieval-Augmented LLMs for Tourism Domain](https://mlsanity.com/paper/2411.08724/) (2024)
- [Efficiently Estimating Data Efficiency for Language Model Fine-tuning](https://mlsanity.com/paper/2512.24991/) (2025)
- [Latent Knowledge as a Predictor of Fact Acquisition in Fine-Tuned Large Language Models](https://mlsanity.com/paper/2601.18468/) (2026)
- [Adaptive Data Augmentation with Multi-armed Bandit: Sample-Efficient Embedding Calibration for Implicit Pattern Recognition](https://mlsanity.com/paper/2602.19385/) (2026)
- [Domain-Adapted Retrieval for In-Context Annotation of Pedagogical Dialogue Acts](https://mlsanity.com/paper/2604.03127/) (2026)

## Abstract (arXiv)

Retrieval augmented generation (RAG) exhibits outstanding performance in promoting the knowledge capabilities of large language models (LLMs) with retrieved documents related to user queries. However, RAG only focuses on improving the response quality of LLMs via enhancing queries indiscriminately with retrieved information, paying little attention to what type of knowledge LLMs really need to answer original queries more accurately. In this paper, we suggest that long-tail knowledge is crucial for RAG as LLMs have already remembered common world knowledge during large-scale pre-training. Based on our observation, we propose a simple but effective long-tail knowledge detection method for LLMs. Specifically, the novel Generative Expected Calibration Error (GECE) metric is derived to measure the ``long-tailness'' of knowledge based on both statistics and semantics. Hence, we retrieve relevant documents and infuse them into the model for patching knowledge loopholes only when the input query relates to long-tail knowledge. Experiments show that, compared to existing RAG pipelines, our method achieves over 4x speedup in average inference time and consistent performance improvement in downstream tasks.