mlsanityData to 7 Oct 2026

kNN-Adapter: Efficient Domain Adaptation for Black-Box Language Models

arXiv 2302.10879 ↗First posted 21 Feb 2023cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Fine-tuning a language model on a new domain is standard practice for domain adaptation. However, it can be infeasible when it comes to modern large-scale language models such as GPT-3, which can only be accessed through APIs, making it difficult to access the internal parameters of the model. In this paper, we propose kNN-Adapter, a method to effectively adapt these black-box large language models (LLMs) to a new domain. The kNN-Adapter builds on top of the retrieval-augmented language model, and adaptively learns to interpolate the output of the language model with retrieval results from a datastore consisting of the target domain data. Our experiments on four different domains demonstrate that kNN-Adapter significantly improves perplexity, and works particularly well in settings with limited access to LLMs. Additionally, we show that kNN-Adapter is more effective than fine-tuning when the amount of training data is limited. We also release a dataset to encourage further study.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 24 of its 29 references that are in the map. First posted on arXiv 21 Feb 2023.

Builds on

24 of 29 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2017Billion-scale similarity search with GPUscited by 2,477
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  8. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  9. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  10. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  11. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  12. 2016Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filteringcited by 517
12 more
  1. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  2. 2022Training Language Models with Memory Augmentationcited by 112
  3. 2021Efficient Nearest Neighbor Language Modelscited by 93
  4. 2021Adaptive Semiparametric Language Modelscited by 83
  5. 2022Neuro-Symbolic Language Modeling with Automaton-augmented Retrievalcited by 60
  6. 2021Adaptive Nearest Neighbor Machine Translationcited by 58
  7. 2017Dynamic Evaluation of Neural Sequence Modelscited by 53
  8. 2021Mention Memory: incorporating textual knowledge into Transformers through entity mention attentioncited by 42
  9. 2022Nonparametric Masked Language Modelingcited by 39
  10. 2021GNN-LM: Language Modeling based on Global Contexts via GNNcited by 27
  11. 2022kNN-Prompt: Nearest Neighbor Zero-Shot Inferencecited by 27
  12. 2022Meta-Learning Fast Weight Language Modelscited by 20

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 7; 2025: 2.

13 papers.

Sort:

By area:

  1. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  2. 2024Retrieval-Augmented Generation for Natural Language Processing: A Surveycited by 70
  3. 2024HYDRA: Model Factorization Framework for Black-Box LLM Personalizationcited by 59
  4. 2024MedAdapter: Efficient Test-Time Adaptation of Large Language Models towards Medical Reasoningcited by 35
  5. 2023CombLM: Adapting Black-Box Language Models through Small Fine-Tuned Modelscited by 26
  6. 2023Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Modelscited by 23
  7. 2023Retrieval-Pretrained Transformer: Long-range Language Modeling with Self-retrievalcited by 20
  8. 2024BBox-Adapter: Lightweight Adapting for Black-Box Large Language Modelscited by 11
  9. 2024Matryoshka Pilot: Learning to Drive Black-Box LLMs with LLMscited by 9
  10. 2024PortLLM: Personalizing Evolving Large Language Models with Training-Free and Portable Model Patchescited by 7
  11. 2024Multi-Prompting Decoder Helps Better Language Understandingcited by 2
  12. 2025Minifinetuning: Low-Data Generation Domain Adaptation through Corrective Self-Distillationcited by 1
1 more
  1. 2025Graph Memory: A Structured and Interpretable Framework for Modality-Agnostic Embedding-Based Inferencecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024RE-Adapt: Reverse Engineered Adaptation of Large Language Models
  2. 2025Mining Useful General Data for Low-Resource Domain Adaptation
  3. 2025Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of Small Multilingual Language Models for Low-Resource Languages
  4. 2024BLADE: Enhancing Black-box Large Language Models with Small Domain-Specific Models
  5. 2024Scalable Multi-Domain Adaptation of Language Models using Modular Experts
  6. 2024Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass
  7. 2023Adapt in Contexts: Retrieval-Augmented Domain Adaptation via In-Context Learning
  8. 2025Franken-Adapter: Cross-Lingual Adaptation of LLMs by Embedding Surgery