mlsanityData to 7 Oct 2026

Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Data

arXiv 2203.08773 ↗First posted 16 Mar 2022cs.CL cs.AI cs.IRcited by 61

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Retrieval-based methods have been shown to be effective in NLP tasks via introducing external knowledge. However, the indexing and retrieving of large-scale corpora bring considerable computational cost. Surprisingly, we found that REtrieving from the traINing datA (REINA) only can lead to significant gains on multiple NLG and NLU tasks. We retrieve the labeled training instances most similar to the input text and then concatenate them with the input to feed into the model to generate the output. Experimental results show that this simple method can achieve significantly better performance on a variety of NLU and NLG tasks, including summarization, machine translation, language modeling, and question answering tasks. For instance, our proposed method achieved state-of-the-art results on XSum, BigPatent, and CommonsenseQA. Our code is released, https://github.com/microsoft/REINA .

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 5 of its 46 references that are in the map. First posted on arXiv 16 Mar 2022.

Builds on

5 of 46 references.

References · 5

  1. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  2. 2018WikiHow: A Large Scale Text Summarization Datasetcited by 203
  3. 2021KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answeringcited by 66
  4. 2021RetGen: A Joint framework for Retrieval and Grounded Text Generation Modelingcited by 22
  5. 2021RetrievalSum: A Retrieval Enhanced Framework for Abstractive Summarizationcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 15; 2023: 23; 2024: 14; 2025: 8; 2026: 1.

61 papers.

Sort:

By area:

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2022In-context Examples Selection for Machine Translationcited by 172
  3. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 136
  4. 2024xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Tokencited by 123
  5. 2023Lift Yourself Up: Retrieval-augmented Text Generation with Self Memorycited by 103
  6. 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
  7. 2023Test-Time Training on Nearest Neighbors for Large Language Modelscited by 82
  8. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  9. 2023TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023cited by 73
  10. 2024Retrieval-Augmented Generation for Natural Language Processing: A Surveycited by 70
  11. 2023kNN Prompting: Beyond-Context Learning with Calibration-Free Nearest Neighbor Inferencecited by 47
  12. 2022Unsupervised Cross-Task Generalization via Retrieval Augmentationcited by 42
38 more
  1. 2024M-RAG: Reinforcing Large Language Model Performance through Retrieval-Augmented Generation with Multiple Partitionscited by 38
  2. 2023Retrieval-Enhanced Contrastive Vision-Text Modelscited by 36
  3. 2024Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustmentcited by 29
  4. 2022Named Entity and Relation Extraction with Multi-Modal Retrievalcited by 26
  5. 2024Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyondcited by 22
  6. 2023Selective Demonstrations for Cross-domain Text-to-SQLcited by 21
  7. 2023Improving Image Recognition by Retrieving from Web-Scale Image-Text Datacited by 20
  8. 2024kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabulariescited by 20
  9. 2022Cross-Lingual Retrieval Augmented Prompt for Low-Resource Languagescited by 17
  10. 2025Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personascited by 17
  11. 2022Generate-and-Retrieve: use your predictions to improve retrieval for semantic parsingcited by 15
  12. 2022G-MAP: General Memory-Augmented Pre-trained Language Model for Domain Taskscited by 9
  13. 2023Pre-Trained Language-Meaning Models for Multilingual Parsing and Generationcited by 9
  14. 2022Zemi: Learning Zero-Shot Semi-Parametric Language Models from Multiple Taskscited by 8
  15. 2023Expository Text Generation: Imitate, Retrieve, Paraphrasecited by 8
  16. 2025A Survey on Retrieval And Structuring Augmented Generation with Large Language Modelscited by 8
  17. 2022Controlling Styles in Neural Machine Translation with Activation Promptcited by 7
  18. 2022Schema-aware Reference as Prompt Improves Data-Efficient Knowledge Graph Constructioncited by 6
  19. 2024Enhancing In-Context Learning via Implicit Demonstration Augmentationcited by 6
  20. 2024ImageRAG: Enhancing Ultra High Resolution Remote Sensing Imagery Analysis with ImageRAGcited by 6
  21. 2022Generalization Properties of Retrieval-based Modelscited by 5
  22. 2025Provoking Multi-modal Few-Shot LVLM via Exploration-Exploitation In-Context Learningcited by 5
  23. 2025SecureReviewer: Enhancing Large Language Models for Secure Code Review through Secure-aware Fine-tuningcited by 5
  24. 2023Prompting Vision Language Model with Knowledge from Large Language Model for Knowledge-Based VQAcited by 4
  25. 2023Improving Language Models Meaning Understanding and Consistency by Learning Conceptual Roles from Dictionarycited by 4
  26. 2024Automatic Combination of Sample Selection Strategies for Few-Shot Learningcited by 4
  27. 2024RAEE: A Robust Retrieval-Augmented Early Exit Framework for Efficient Inferencecited by 4
  28. 2024DEGAP: Dual Event-Guided Adaptive Prefixes for Templated-Based Event Argument Extraction with Slot Queryingcited by 3
  29. 2024Improving Zero-shot LLM Re-Ranker with Risk Minimizationcited by 3
  30. 2023Retrieval-Augmented Meta Learning for Low-Resource Text Classificationcited by 2
  31. 2023RIGHT: Retrieval-augmented Generation for Mainstream Hashtag Recommendationcited by 2
  32. 2024A Statistical Framework for Data-dependent Retrieval-Augmented Modelscited by 2
  33. 2022Self-Adaptive Named Entity Recognition by Retrieving Unstructured Knowledgecited by 1
  34. 2022A Unified Knowledge Graph Augmentation Service for Boosting Domain-specific NLP Taskscited by 1
  35. 2023Cross-Model Comparative Loss for Enhancing Neuronal Utility in Language Understandingcited by 1
  36. 2023Shuo Wen Jie Zi: Rethinking Dictionaries and Glyphs for Chinese Language Pre-trainingcited by 1
  37. 2023Rethinking Translation Memory Augmented Neural Machine Translationcited by 1
  38. 2023Prompt Guided Copy Mechanism for Conversational Question Answeringcited by 1

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020REALM: Retrieval-Augmented Language Model Pre-Training
  2. 2023Improving Language Models via Plug-and-Play Retrieval Feedback
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  4. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Context
  5. 2024Retrieval-Augmented Data Augmentation for Low-Resource Domain Tasks
  6. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP
  7. 2024Training Task Experts through Retrieval Based Distillation
  8. 2024Unsupervised Information Refinement Training of Large Language Models for Retrieval-Augmented Generation