mlsanityData to 7 Oct 2026

BERT-kNN: Adding a kNN Search Component to Pretrained Language Models for Better QA

arXiv 2005.00766 ↗First posted 2 May 2020cs.CLcited by 36

Not on the map. Why?

Abstract (arXiv)

Khandelwal et al. (2020) use a k-nearest-neighbor (kNN) component to improve language model performance. We show that this idea is beneficial for open-domain question answering (QA). To improve the recall of facts encountered during training, we combine BERT (Devlin et al., 2019) with a traditional information retrieval step (IR) and a kNN search over a large datastore of an embedded text collection. Our contributions are as follows: i) BERT-kNN outperforms BERT on cloze-style QA by large margins without any further training. ii) We show that BERT often identifies the correct response category (e.g., US city), but only kNN recovers the factually correct answer (e.g., "Miami"). iii) Compared to BERT, BERT-kNN excels for rare facts. iv) BERT-kNN can easily handle facts not covered by BERT's training set, e.g., recent events.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 12; 2023: 7; 2024: 3; 2025: 2; 2026: 1.

36 papers.

Sort:

By area:

  1. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  2. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 251
  3. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  4. 2021Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Basescited by 107
  5. 2021Adaptive Semiparametric Language Modelscited by 83
  6. 2021Adaptive Nearest Neighbor Machine Translationcited by 58
  7. 2022Retrieval-Enhanced Machine Learningcited by 46
  8. 2022Nonparametric Masked Language Modelingcited by 39
  9. 2022Decoupling Knowledge from Memorization: Retrieval-augmented Prompt Learningcited by 36
  10. 2021ConvFiT: Conversational Fine-Tuning of Pretrained Language Modelscited by 25
  11. 2021DREAM: Improving Situational QA by First Elaborating the Situationcited by 18
  12. 2022You can't pick your neighbors, or can you? When and how to rely on retrieval in the kNN-LMcited by 18
24 more
  1. 2022Relation Extraction as Open-book Examination: Retrieval-enhanced Prompt Tuningcited by 13
  2. 2022Reasoning Through Memorization: Nearest Neighbor Knowledge Graph Embeddingscited by 12
  3. 2021Static Embeddings as Efficient Knowledge Bases?cited by 11
  4. 2024Retrieval-Enhanced Machine Learning: Synthesis and Opportunitiescited by 11
  5. 2025Pre-training Limited Memory Language Models with Internal and External Knowledgecited by 11
  6. 2025Defending Against Knowledge Poisoning Attacks During Retrieval-Augmented Generationcited by 11
  7. 2021Not Far Away, Not So Close: Sample Efficient Nearest Neighbour Data Augmentation via MiniMaxcited by 10
  8. 2023Legal Question-Answering in the Indian Context: Efficacy, Challenges, and Potential of Modern AI Modelscited by 10
  9. 2021KNN-BERT: Fine-Tuning Pre-Trained Models with KNN Classifiercited by 7
  10. 2023Retrieval-Enhanced Visual Prompt Learning for Few-shot Classificationcited by 6
  11. 2024ARAIDA: Analogical Reasoning-Augmented Interactive Data Annotationcited by 6
  12. 2022Error-Robust Retrieval for Chinese Spelling Checkcited by 4
  13. 2023Mitigating Catastrophic Forgetting in Task-Incremental Continual Learning with Adaptive Classification Criterioncited by 4
  14. 2021A Neighbourhood Framework for Resource-Lean Content Flaggingcited by 3
  15. 2022WikiOmnia: generative QA corpus on the whole Russian Wikipediacited by 2
  16. 2022Augmentation with Projection: Towards an Effective and Efficient Data Augmentation Paradigm for Distillationcited by 2
  17. 2023Machine Reading Comprehension using Case-based Reasoningcited by 2
  18. 2023RSpell: Retrieval-augmented Framework for Domain Adaptive Chinese Spelling Checkcited by 2
  19. 2023Towards Faster k-Nearest-Neighbor Machine Translationcited by 1
  20. 2022Transferring Knowledge via Neighborhood-Aware Optimal Transport for Low-Resource Hate Speech Detectioncited by 0
  21. 2022Reranking Overgenerated Responses for End-to-End Task-Oriented Dialogue Systemscited by 0
  22. 2022Task-Specific Embeddings for Ante-Hoc Explainable Text Classificationcited by 0
  23. 2023Revisiting k-NN for Fine-tuning Pre-trained Language Modelscited by 0
  24. 2026AILQA: Evaluating AI-Driven Legal Question Answering Systems for the Indian Legal Systemcited by 0