# kNN-Adapter: Efficient Domain Adaptation for Black-Box Language Models

arXiv: https://arxiv.org/abs/2302.10879 · HTML: https://mlsanity.com/paper/2302.10879/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 29 references that are in the map.
- First posted on arXiv 21 Feb 2023.

## Builds on (most cited first, 24 of 24)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [Retrieval-Augmented Multimodal Language Modeling](https://mlsanity.com/paper/2211.12561/) (2022)
- [Training Language Models with Memory Augmentation](https://mlsanity.com/paper/2205.12674/) (2022)
- [Efficient Nearest Neighbor Language Models](https://mlsanity.com/paper/2109.04212/) (2021)
- [Adaptive Semiparametric Language Models](https://mlsanity.com/paper/2102.02557/) (2021)
- [Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval](https://mlsanity.com/paper/2201.12431/) (2022)
- [Adaptive Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2105.13022/) (2021)
- [Dynamic Evaluation of Neural Sequence Models](https://mlsanity.com/paper/1709.07432/) (2017)
- [Mention Memory: incorporating textual knowledge into Transformers through entity mention attention](https://mlsanity.com/paper/2110.06176/) (2021)
- [Nonparametric Masked Language Modeling](https://mlsanity.com/paper/2212.01349/) (2022)
- [GNN-LM: Language Modeling based on Global Contexts via GNN](https://mlsanity.com/paper/2110.08743/) (2021)
- [kNN-Prompt: Nearest Neighbor Zero-Shot Inference](https://mlsanity.com/paper/2205.13792/) (2022)
- [Meta-Learning Fast Weight Language Models](https://mlsanity.com/paper/2212.02475/) (2022)

## Built on by (most cited first, 14 of 14)

- [SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore](https://mlsanity.com/paper/2308.04430/) (2023)
- [Retrieval-Augmented Generation for Natural Language Processing: A Survey](https://mlsanity.com/paper/2407.13193/) (2024)
- [HYDRA: Model Factorization Framework for Black-Box LLM Personalization](https://mlsanity.com/paper/2406.02888/) (2024)
- [MedAdapter: Efficient Test-Time Adaptation of Large Language Models towards Medical Reasoning](https://mlsanity.com/paper/2405.03000/) (2024)
- [CombLM: Adapting Black-Box Language Models through Small Fine-Tuned Models](https://mlsanity.com/paper/2305.16876/) (2023)
- [Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Models](https://mlsanity.com/paper/2310.07589/) (2023)
- [Retrieval-Pretrained Transformer: Long-range Language Modeling with Self-retrieval](https://mlsanity.com/paper/2306.13421/) (2023)
- [BBox-Adapter: Lightweight Adapting for Black-Box Large Language Models](https://mlsanity.com/paper/2402.08219/) (2024)
- [Matryoshka Pilot: Learning to Drive Black-Box LLMs with LLMs](https://mlsanity.com/paper/2410.20749/) (2024)
- [PortLLM: Personalizing Evolving Large Language Models with Training-Free and Portable Model Patches](https://mlsanity.com/paper/2410.10870/) (2024)
- [Multi-Prompting Decoder Helps Better Language Understanding](https://mlsanity.com/paper/2406.06279/) (2024)
- [Minifinetuning: Low-Data Generation Domain Adaptation through Corrective Self-Distillation](https://mlsanity.com/paper/2506.15702/) (2025)
- [Graph Memory: A Structured and Interpretable Framework for Modality-Agnostic Embedding-Based Inference](https://mlsanity.com/paper/2511.14961/) (2025)
- [kNNProxy: Efficient Training-Free Proxy Alignment for Black-Box Zero-Shot LLM-Generated Text Detection](https://mlsanity.com/paper/2604.02008/) (2026)

## Abstract (arXiv)

Fine-tuning a language model on a new domain is standard practice for domain adaptation. However, it can be infeasible when it comes to modern large-scale language models such as GPT-3, which can only be accessed through APIs, making it difficult to access the internal parameters of the model. In this paper, we propose kNN-Adapter, a method to effectively adapt these black-box large language models (LLMs) to a new domain. The kNN-Adapter builds on top of the retrieval-augmented language model, and adaptively learns to interpolate the output of the language model with retrieval results from a datastore consisting of the target domain data. Our experiments on four different domains demonstrate that kNN-Adapter significantly improves perplexity, and works particularly well in settings with limited access to LLMs. Additionally, we show that kNN-Adapter is more effective than fine-tuning when the amount of training data is limited. We also release a dataset to encourage further study.