# Nearest Neighbor Language Models for Stylistic Controllable Generation

arXiv: https://arxiv.org/abs/2210.15762 · HTML: https://mlsanity.com/paper/2210.15762/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 23 references that are in the map.
- First posted on arXiv 27 Oct 2022.

## Builds on (most cited first, 19 of 19)

- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts](https://mlsanity.com/paper/2105.03023/) (2021)
- [Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification](https://mlsanity.com/paper/1903.04561/) (2019)
- [FUDGE: Controlled Text Generation With Future Discriminators](https://mlsanity.com/paper/2104.05218/) (2021)
- [Memorizing Transformers](https://mlsanity.com/paper/2203.08913/) (2022)
- [Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2010.00710/) (2020)
- [A Computational Approach to Politeness with Application to Social Factors](https://mlsanity.com/paper/1306.6078/) (2013)
- [Efficient Nearest Neighbor Language Models](https://mlsanity.com/paper/2109.04212/) (2021)
- [Adaptive Semiparametric Language Models](https://mlsanity.com/paper/2102.02557/) (2021)
- [Improving Neural Language Models with a Continuous Cache](https://mlsanity.com/paper/1612.04426/) (2016)
- [Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval](https://mlsanity.com/paper/2201.12431/) (2022)
- [Unbounded cache model for online language modeling with open vocabulary](https://mlsanity.com/paper/1711.02604/) (2017)
- [Fast Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2105.14528/) (2021)
- [Capturing Structural Locality in Non-parametric Language Models](https://mlsanity.com/paper/2110.02870/) (2021)
- [Faster Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2112.08152/) (2021)

## Built on by (most cited first, 2 of 2)

- [Learn to Memorize: Scalable Continual Learning in Semiparametric Models with Mixture-of-Neighbors Induction Memory](https://mlsanity.com/paper/2303.01421/) (2023)
- [Style Locality for Controllable Generation with kNN Language Models](https://mlsanity.com/paper/2311.00475/) (2023)

## Abstract (arXiv)

Recent language modeling performance has been greatly improved by the use of external memory. This memory encodes the context so that similar contexts can be recalled during decoding. This similarity depends on how the model learns to encode context, which can be altered to include other attributes, such as style. We construct and evaluate an architecture for this purpose, using corpora annotated for politeness, formality, and toxicity. Through extensive experiments and human evaluation we demonstrate the potential of our method to generate text while controlling style. We find that style-specific datastores improve generation performance, though results vary greatly across styles, and the effect of pretraining data and specific styles should be explored in future work.