# Improving Retrieval-Augmented Large Language Models via Data Importance Learning

arXiv: https://arxiv.org/abs/2307.03027 · HTML: https://mlsanity.com/paper/2307.03027/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation.
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 6 Jul 2023.

## Placed in

- Retrieval-augmented generation / [Generation and context integration](https://mlsanity.com/rag/generation-and-context-integration/)

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Learning from Noisy Labels with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2007.08199/) (2020)
- [Can Foundation Models Wrangle Your Data?](https://mlsanity.com/paper/2205.09911/) (2022)
- [Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering](https://mlsanity.com/paper/2210.02627/) (2022)
- [The Cost of Training NLP Models: A Concise Overview](https://mlsanity.com/paper/2004.08900/) (2020)
- [Decentralized Training of Foundation Models in Heterogeneous Environments](https://mlsanity.com/paper/2206.01288/) (2022)
- [TVM: An Automated End-to-End Optimizing Compiler for Deep Learning](https://mlsanity.com/paper/1802.04799/) (2018)
- [Transcending Scaling Laws with 0.1% Extra Compute](https://mlsanity.com/paper/2210.11399/) (2022)
- [Retrieval-Enhanced Machine Learning](https://mlsanity.com/paper/2205.01230/) (2022)
- [Fine-tuning Pre-Trained Transformer Language Models to Distantly Supervised Relation Extraction](https://mlsanity.com/paper/1906.08646/) (2019)
- [Data Debugging with Shapley Importance over End-to-End Machine Learning Pipelines](https://mlsanity.com/paper/2204.11131/) (2022)
- [A Note on "Efficient Task-Specific Data Valuation for Nearest Neighbor Algorithms"](https://mlsanity.com/paper/2304.04258/) (2023)

## Built on by (most cited first, 17 of 17)

- [Large Language Models for Information Retrieval: A Survey](https://mlsanity.com/paper/2308.07107/) (2023)
- [PaperQA: Retrieval-Augmented Generative Agent for Scientific Research](https://mlsanity.com/paper/2312.07559/) (2023)
- [R-Tuning: Instructing Large Language Models to Say `I Don't Know'](https://mlsanity.com/paper/2311.09677/) (2023)
- [A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Models](https://mlsanity.com/paper/2501.13958/) (2025)
- [Retrieval-Augmented Generation for Natural Language Processing: A Survey](https://mlsanity.com/paper/2407.13193/) (2024)
- [A Survey on Knowledge-Oriented Retrieval-Augmented Generation](https://mlsanity.com/paper/2503.10677/) (2025)
- [Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Models](https://mlsanity.com/paper/2407.08039/) (2024)
- [Building Trustworthy NeuroSymbolic AI Systems: Consistency, Reliability, Explainability, and Safety](https://mlsanity.com/paper/2312.06798/) (2023)
- [Enhancing textual textbook question answering with large language models and retrieval augmented generation](https://mlsanity.com/paper/2402.05128/) (2024)
- [Retrieval-Enhanced Machine Learning: Synthesis and Opportunities](https://mlsanity.com/paper/2407.12982/) (2024)
- [Semantic Self-Consistency: Enhancing Language Model Reasoning via Semantic Weighting](https://mlsanity.com/paper/2410.07839/) (2024)
- [Furthest Reasoning with Plan Assessment: Stable Reasoning Path with Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2309.12767/) (2023)
- [SparkRA: A Retrieval-Augmented Knowledge Service System Based on Spark Large Language Model](https://mlsanity.com/paper/2408.06574/) (2024)
- [Beyond Independent Passages: Adaptive Passage Combination Retrieval for Retrieval Augmented Open-Domain Question Answering](https://mlsanity.com/paper/2507.04069/) (2025)
- [Detrimental Contexts in Open-Domain Question Answering](https://mlsanity.com/paper/2310.18077/) (2023)
- [Beyond Retrieval: Joint Supervision and Multimodal Document Ranking for Textbook Question Answering](https://mlsanity.com/paper/2505.13520/) (2025)
- [AI-Driven Generation of Old English: A Framework for Low-Resource Languages](https://mlsanity.com/paper/2507.20111/) (2025)

## Abstract (arXiv)

Retrieval augmentation enables large language models to take advantage of external knowledge, for example on tasks like question answering and data imputation. However, the performance of such retrieval-augmented models is limited by the data quality of their underlying retrieval corpus. In this paper, we propose an algorithm based on multilinear extension for evaluating the data importance of retrieved data points. There are exponentially many terms in the multilinear extension, and one key contribution of this paper is a polynomial time algorithm that computes exactly, given a retrieval-augmented model with an additive utility function and a validation set, the data importance of data points in the retrieval corpus using the multilinear extension of the model's utility function. We further proposed an even more efficient (ε, δ)-approximation algorithm. Our experimental results illustrate that we can enhance the performance of large language models by only pruning or reweighting the retrieval corpus, without requiring further training. For some tasks, this even allows a small model (e.g., GPT-JT), augmented with a search engine API, to outperform GPT-3.5 (without retrieval augmentation). Moreover, we show that weights based on multilinear extension can be computed efficiently in practice (e.g., in less than ten minutes for a corpus with 100 million elements).