# Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Data

arXiv: https://arxiv.org/abs/2203.08773 · HTML: https://mlsanity.com/paper/2203.08773/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 5 of its 46 references that are in the map.
- First posted on arXiv 16 Mar 2022.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 5 of 5)

- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [WikiHow: A Large Scale Text Summarization Dataset](https://mlsanity.com/paper/1810.09305/) (2018)
- [KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answering](https://mlsanity.com/paper/2110.04330/) (2021)
- [RetGen: A Joint framework for Retrieval and Grounded Text Generation Modeling](https://mlsanity.com/paper/2105.06597/) (2021)
- [RetrievalSum: A Retrieval Enhanced Framework for Abstractive Summarization](https://mlsanity.com/paper/2109.07943/) (2021)

## Built on by (most cited first, 30 of 61)

- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [In-context Examples Selection for Machine Translation](https://mlsanity.com/paper/2212.02437/) (2022)
- [Diverse Demonstrations Improve In-context Compositional Generalization](https://mlsanity.com/paper/2212.06800/) (2022)
- [xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token](https://mlsanity.com/paper/2405.13792/) (2024)
- [Lift Yourself Up: Retrieval-augmented Text Generation with Self Memory](https://mlsanity.com/paper/2305.02437/) (2023)
- [Test-Time Training on Nearest Neighbors for Large Language Models](https://mlsanity.com/paper/2305.18466/) (2023)
- [SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentation](https://mlsanity.com/paper/2209.15323/) (2022)
- [A Survey of Knowledge Enhanced Pre-trained Language Models](https://mlsanity.com/paper/2211.05994/) (2022)
- [TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023](https://mlsanity.com/paper/2307.14338/) (2023)
- [Retrieval-Augmented Generation for Natural Language Processing: A Survey](https://mlsanity.com/paper/2407.13193/) (2024)
- [kNN Prompting: Beyond-Context Learning with Calibration-Free Nearest Neighbor Inference](https://mlsanity.com/paper/2303.13824/) (2023)
- [Unsupervised Cross-Task Generalization via Retrieval Augmentation](https://mlsanity.com/paper/2204.07937/) (2022)
- [M-RAG: Reinforcing Large Language Model Performance through Retrieval-Augmented Generation with Multiple Partitions](https://mlsanity.com/paper/2405.16420/) (2024)
- [Retrieval-Enhanced Contrastive Vision-Text Models](https://mlsanity.com/paper/2306.07196/) (2023)
- [Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustment](https://mlsanity.com/paper/2403.02738/) (2024)
- [Named Entity and Relation Extraction with Multi-Modal Retrieval](https://mlsanity.com/paper/2212.01612/) (2022)
- [Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyond](https://mlsanity.com/paper/2407.10005/) (2024)
- [Selective Demonstrations for Cross-domain Text-to-SQL](https://mlsanity.com/paper/2310.06302/) (2023)
- [Improving Image Recognition by Retrieving from Web-Scale Image-Text Data](https://mlsanity.com/paper/2304.05173/) (2023)
- [kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabularies](https://mlsanity.com/paper/2404.09447/) (2024)
- [Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personas](https://mlsanity.com/paper/2501.11549/) (2025)
- [Cross-Lingual Retrieval Augmented Prompt for Low-Resource Languages](https://mlsanity.com/paper/2212.09651/) (2022)
- [Generate-and-Retrieve: use your predictions to improve retrieval for semantic parsing](https://mlsanity.com/paper/2209.14899/) (2022)
- [Pre-Trained Language-Meaning Models for Multilingual Parsing and Generation](https://mlsanity.com/paper/2306.00124/) (2023)
- [G-MAP: General Memory-Augmented Pre-trained Language Model for Domain Tasks](https://mlsanity.com/paper/2212.03613/) (2022)
- [Zemi: Learning Zero-Shot Semi-Parametric Language Models from Multiple Tasks](https://mlsanity.com/paper/2210.00185/) (2022)
- [Expository Text Generation: Imitate, Retrieve, Paraphrase](https://mlsanity.com/paper/2305.03276/) (2023)
- [A Survey on Retrieval And Structuring Augmented Generation with Large Language Models](https://mlsanity.com/paper/2509.10697/) (2025)
- [Controlling Styles in Neural Machine Translation with Activation Prompt](https://mlsanity.com/paper/2212.08909/) (2022)
- [Provoking Multi-modal Few-Shot LVLM via Exploration-Exploitation In-Context Learning](https://mlsanity.com/paper/2506.09473/) (2025)

## Abstract (arXiv)

Retrieval-based methods have been shown to be effective in NLP tasks via introducing external knowledge. However, the indexing and retrieving of large-scale corpora bring considerable computational cost. Surprisingly, we found that REtrieving from the traINing datA (REINA) only can lead to significant gains on multiple NLG and NLU tasks. We retrieve the labeled training instances most similar to the input text and then concatenate them with the input to feed into the model to generate the output. Experimental results show that this simple method can achieve significantly better performance on a variety of NLU and NLG tasks, including summarization, machine translation, language modeling, and question answering tasks. For instance, our proposed method achieved state-of-the-art results on XSum, BigPatent, and CommonsenseQA. Our code is released, https://github.com/microsoft/REINA .