# Unbounded cache model for online language modeling with open vocabulary

arXiv: https://arxiv.org/abs/1711.02604 · HTML: https://mlsanity.com/paper/1711.02604/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Nov 2017.

## Built on by (most cited first, 30 of 37)

- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling](https://mlsanity.com/paper/2111.03930/) (2021)
- [Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification](https://mlsanity.com/paper/2207.09519/) (2022)
- [Efficient Test-Time Adaptation of Vision-Language Models](https://mlsanity.com/paper/2403.18293/) (2024)
- [Training Language Models with Memory Augmentation](https://mlsanity.com/paper/2205.12674/) (2022)
- [Adaptive Semiparametric Language Models](https://mlsanity.com/paper/2102.02557/) (2021)
- [Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval](https://mlsanity.com/paper/2205.13760/) (2022)
- [Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval](https://mlsanity.com/paper/2201.12431/) (2022)
- [Larimar: Large Language Models with Episodic Memory Control](https://mlsanity.com/paper/2403.11901/) (2024)
- [Decoupling Knowledge from Memorization: Retrieval-augmented Prompt Learning](https://mlsanity.com/paper/2205.14704/) (2022)
- [Why do Nearest Neighbor Language Models Work?](https://mlsanity.com/paper/2301.02828/) (2023)
- [A Survey on Open Set Recognition](https://mlsanity.com/paper/2109.00893/) (2021)
- [Efficient Prompt Caching via Embedding Similarity](https://mlsanity.com/paper/2402.01173/) (2024)
- [Capturing Structural Locality in Non-parametric Language Models](https://mlsanity.com/paper/2110.02870/) (2021)
- [Rethinking Nearest Neighbors for Visual Classification](https://mlsanity.com/paper/2112.08459/) (2021)
- [HistAlign: Improving Context Dependency in Language Generation by Aligning with History](https://mlsanity.com/paper/2305.04782/) (2023)
- [RAP: Retrieval-Augmented Planner for Adaptive Procedure Planning in Instructional Videos](https://mlsanity.com/paper/2403.18600/) (2024)
- [Nearest Neighbors Meet Deep Neural Networks for Point Cloud Analysis](https://mlsanity.com/paper/2303.00703/) (2023)
- [Artefact Retrieval: Overview of NLP Models with Knowledge Base Access](https://mlsanity.com/paper/2201.09651/) (2022)
- [kNN-Embed: Locally Smoothed Embedding Mixtures For Multi-interest Candidate Retrieval](https://mlsanity.com/paper/2205.06205/) (2022)
- [Point-Cache: Test-time Dynamic and Hierarchical Cache for Robust and Generalizable Point Cloud Analysis](https://mlsanity.com/paper/2503.12150/) (2025)
- [Protriever: End-to-End Differentiable Protein Homology Search for Fitness Prediction](https://mlsanity.com/paper/2506.08954/) (2025)
- [Unified modality separation: A vision-language framework for unsupervised domain adaptation](https://mlsanity.com/paper/2508.04987/) (2025)
- [ETTA: Efficient Test-Time Adaptation for Vision-Language Models through Dynamic Embedding Updates](https://mlsanity.com/paper/2508.05898/) (2025)
- [Nearest Neighbor Language Models for Stylistic Controllable Generation](https://mlsanity.com/paper/2210.15762/) (2022)
- [Towards Continual Entity Learning in Language Models for Conversational Agents](https://mlsanity.com/paper/2108.00082/) (2021)
- [N-Gram Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2301.12866/) (2023)
- [Long-Tail Crisis in Nearest Neighbor Language Models](https://mlsanity.com/paper/2503.22426/) (2025)
- [Plug-and-play Class-aware Knowledge Injection for Prompt Learning with Visual-Language Model](https://mlsanity.com/paper/2605.05910/) (2026)

## Abstract (arXiv)

Recently, continuous cache models were proposed as extensions to recurrent neural network language models, to adapt their predictions to local changes in the data distribution. These models only capture the local context, of up to a few thousands tokens. In this paper, we propose an extension of continuous cache models, which can scale to larger contexts. In particular, we use a large scale non-parametric memory component that stores all the hidden activations seen in the past. We leverage recent advances in approximate nearest neighbor search and quantization algorithms to store millions of representations while searching them efficiently. We conduct extensive experiments showing that our approach significantly improves the perplexity of pre-trained language models on new distributions, and can scale efficiently to much larger contexts than previously proposed local cache models.