mlsanityData to 7 Oct 2026

kNN-Prompt: Nearest Neighbor Zero-Shot Inference

arXiv 2205.13792 ↗First posted 27 May 2022cs.CLcited by 27

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Retrieval-augmented language models (LMs) use non-parametric memory to substantially outperform their non-retrieval counterparts on perplexity-based evaluations, but it is an open question whether they achieve similar gains in few- and zero-shot end-task accuracy. We extensively study one such model, the k-nearest neighbor LM (kNN-LM), showing that the gains marginally transfer. The main challenge is to achieve coverage of the verbalizer tokens that define the different end-task class labels. To address this challenge, we also introduce kNN-Prompt, a simple and effective kNN-LM with automatically expanded fuzzy verbalizers (e.g. to expand terrible to also include silly and other task-specific synonyms for sentiment classification). Across nine diverse end-tasks, using kNN-Prompt with GPT-2 large yields significant performance boosts over strong zero-shot baselines (13.4% absolute improvement over the base LM on average). We also show that other advantages of non-parametric augmentation hold for end tasks; kNN-Prompt is effective for domain adaptation with no further training, and gains increase with the size of the retrieval model.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 27 of its 39 references that are in the map. First posted on arXiv 27 May 2022.

Builds on

27 of 39 references.

Closest ancestors · 2

Same direction, newest first

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May
  2. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2017Billion-scale similarity search with GPUscited by 2,477
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  7. 2019Language Models as Knowledge Bases?cited by 1,922
  8. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  9. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  10. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  12. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
13 more
  1. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  2. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  3. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  4. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  5. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  6. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  7. 2016Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filteringcited by 517
  8. 2021MetaICL: Learning to Learn In Contextcited by 450
  9. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  10. 2020Nearest Neighbor Machine Translationcited by 209
  11. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  12. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  13. 2021Efficient Nearest Neighbor Language Modelscited by 93

Built on by

Cited by year

Citing papers per year: 2022: 6; 2023: 12; 2024: 6; 2025: 3.

27 papers.

Sort:

By area:

  1. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  2. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  3. 2023Text Classification via Large Language Modelscited by 104
  4. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  5. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  6. 2023Self-Knowledge Guided Retrieval Augmentation for Large Language Modelscited by 69
  7. 2022TIARA: Multi-grained Retrieval for Robust Question Answering over Large Knowledge Basescited by 59
  8. 2023kNN Prompting: Beyond-Context Learning with Calibration-Free Nearest Neighbor Inferencecited by 47
  9. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46
  10. 2022Nonparametric Masked Language Modelingcited by 39
  11. 2023RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Modelscited by 36
  12. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
15 more
  1. 2022Data-Efficient Finetuning Using Cross-Task Nearest Neighborscited by 26
  2. 2023ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrievalcited by 26
  3. 2022Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learnerscited by 22
  4. 2025MLP Memory: A Retriever-Pretrained Memory for Large Language Modelscited by 19
  5. 2025Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Modelscited by 16
  6. 2023kNN-Adapter: Efficient Domain Adaptation for Black-Box Language Modelscited by 13
  7. 2024JoPA:Explaining Large Language Model's Generation via Joint Prompt Attributioncited by 6
  8. 2024Evaluating and Safeguarding the Adversarial Robustness of Retrieval-Based In-Context Learningcited by 4
  9. 2024Superposed Decoding: Multiple Generations from a Single Autoregressive Inference Passcited by 4
  10. 2024Language Modelling Approaches to Adaptive Machine Translationcited by 3
  11. 2022Efficiently Enhancing Zero-Shot Performance of Instruction Following Model via Retrieval of Soft Promptcited by 2
  12. 2023CELDA: Leveraging Black-box Language Model as Enhanced Classifier without Labelscited by 2
  13. 2024R4: Reinforced Retriever-Reorder-Responder for Retrieval-Augmented Large Language Modelscited by 1
  14. 2023Revisiting k-NN for Fine-tuning Pre-trained Language Modelscited by 0
  15. 2025DelvePO: Direction-Guided Self-Evolving Framework for Flexible Prompt Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Great Memory, Shallow Reasoning: Limits of kNN-LMs
  2. 2023Why do Nearest Neighbor Language Models Work?
  3. 2025Long-Tail Crisis in Nearest Neighbor Language Models
  4. 2022Cross-Lingual Retrieval Augmented Prompt for Low-Resource Languages
  5. 2021RETRONLU: Retrieval Augmented Task-Oriented Semantic Parsing
  6. 2022Decoupling Knowledge from Memorization: Retrieval-augmented Prompt Learning
  7. 2024Evaluating Zero-Shot Multilingual Aspect-Based Sentiment Analysis with Large Language Models
  8. 2022Internet-augmented language models through few-shot prompting for open-domain question answering