# Decoupling Knowledge from Memorization: Retrieval-augmented Prompt Learning

arXiv: https://arxiv.org/abs/2205.14704 · HTML: https://mlsanity.com/paper/2205.14704/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation.
- Builds on 51 of its 78 references that are in the map.
- First posted on arXiv 29 May 2022.

## Placed in

- Retrieval-augmented generation / [Generation and context integration](https://mlsanity.com/rag/generation-and-context-integration/)

## Builds on (most cited first, 30 of 51)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](https://mlsanity.com/paper/2008.03703/) (2020)
- [Does Learning Require Memorization? A Short Tale about a Long Tail](https://mlsanity.com/paper/1906.05271/) (2019)
- [PTR: Prompt Tuning with Rules for Text Classification](https://mlsanity.com/paper/2105.11259/) (2021)

## Built on by (most cited first, 30 of 36)

- [A Comprehensive Study of Knowledge Editing for Large Language Models](https://mlsanity.com/paper/2401.01286/) (2024)
- [Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering](https://mlsanity.com/paper/2306.04136/) (2023)
- [A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Models](https://mlsanity.com/paper/2501.13958/) (2025)
- [Dr.ICL: Demonstration-Retrieved In-context Learning](https://mlsanity.com/paper/2305.14128/) (2023)
- [Retrieval-Augmented Generation for Natural Language Processing: A Survey](https://mlsanity.com/paper/2407.13193/) (2024)
- [Generative Knowledge Graph Construction: A Review](https://mlsanity.com/paper/2210.12714/) (2022)
- [kNN Prompting: Beyond-Context Learning with Calibration-Free Nearest Neighbor Inference](https://mlsanity.com/paper/2303.13824/) (2023)
- [Complex Logical Reasoning over Knowledge Graphs using Large Language Models](https://mlsanity.com/paper/2305.01157/) (2023)
- [ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrieval](https://mlsanity.com/paper/2305.10703/) (2023)
- [Towards Unified Multimodal Editing with Enhanced Knowledge Collaboration](https://mlsanity.com/paper/2409.19872/) (2024)
- [PromptIntern: Saving Inference Costs by Internalizing Recurrent Prompt during Large Language Model Fine-tuning](https://mlsanity.com/paper/2407.02211/) (2024)
- [Large Knowledge Model: Perspectives and Challenges](https://mlsanity.com/paper/2312.02706/) (2023)
- [Retrieval-Enhanced Machine Learning: Synthesis and Opportunities](https://mlsanity.com/paper/2407.12982/) (2024)
- [FT2Ra: A Fine-Tuning-Inspired Approach to Retrieval-Augmented Code Completion](https://mlsanity.com/paper/2404.01554/) (2024)
- [ROME: Memorization Insights from Text, Logits and Representation](https://mlsanity.com/paper/2403.00510/) (2024)
- [Towards Building Specialized Generalist AI with System 1 and System 2 Fusion](https://mlsanity.com/paper/2407.08642/) (2024)
- [Train/Test-Time Adaptation with Retrieval](https://mlsanity.com/paper/2303.14333/) (2023)
- [Schema-aware Reference as Prompt Improves Data-Efficient Knowledge Graph Construction](https://mlsanity.com/paper/2210.10709/) (2022)
- [Retrieval-Enhanced Visual Prompt Learning for Few-shot Classification](https://mlsanity.com/paper/2306.02243/) (2023)
- [Retrieval Augmented Deep Anomaly Detection for Tabular Data](https://mlsanity.com/paper/2401.17052/) (2024)
- [From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systems](https://mlsanity.com/paper/2506.04565/) (2025)
- [Towards Realistic Low-resource Relation Extraction: A Benchmark with Empirical Baseline Study](https://mlsanity.com/paper/2210.10678/) (2022)
- [ReFusion: Improving Natural Language Understanding with Computation-Efficient Retrieval Representation Fusion](https://mlsanity.com/paper/2401.02993/) (2024)
- [MixPro: Simple yet Effective Data Augmentation for Prompt-based Learning](https://mlsanity.com/paper/2304.09402/) (2023)
- [SciPrompt: Knowledge-augmented Prompting for Fine-grained Categorization of Scientific Topics](https://mlsanity.com/paper/2410.01946/) (2024)
- [Thinking with Reasoning Skills: Fewer Tokens, More Accuracy](https://mlsanity.com/paper/2604.21764/) (2026)
- [Retrieval-Augmented Meta Learning for Low-Resource Text Classification](https://mlsanity.com/paper/2309.04979/) (2023)
- [A Concept Knowledge Graph for User Next Intent Prediction at Alipay](https://mlsanity.com/paper/2301.00503/) (2023)
- [DRAFT: Dense Retrieval Augmented Few-shot Topic classifier Framework](https://mlsanity.com/paper/2312.02532/) (2023)
- [Research on the Proximity Relationships of Psychosomatic Disease Knowledge Graph Modules Extracted by Large Language Models](https://mlsanity.com/paper/2412.18419/) (2024)

## Abstract (arXiv)

Prompt learning approaches have made waves in natural language processing by inducing better few-shot performance while they still follow a parametric-based learning paradigm; the oblivion and rote memorization problems in learning may encounter unstable generalization issues. Specifically, vanilla prompt learning may struggle to utilize atypical instances by rote during fully-supervised training or overfit shallow patterns with low-shot data. To alleviate such limitations, we develop RetroPrompt with the motivation of decoupling knowledge from memorization to help the model strike a balance between generalization and memorization. In contrast with vanilla prompt learning, RetroPrompt constructs an open-book knowledge-store from training instances and implements a retrieval mechanism during the process of input, training and inference, thus equipping the model with the ability to retrieve related contexts from the training corpus as cues for enhancement. Extensive experiments demonstrate that RetroPrompt can obtain better performance in both few-shot and zero-shot settings. Besides, we further illustrate that our proposed RetroPrompt can yield better generalization abilities with new datasets. Detailed analysis of memorization indeed reveals RetroPrompt can reduce the reliance of language models on memorization; thus, improving generalization for downstream tasks. Code is available in https://github.com/zjunlp/PromptKG/tree/main/research/RetroPrompt.