# Low-Rank RNN Adaptation for Context-Aware Language Modeling

arXiv: https://arxiv.org/abs/1710.02603 · HTML: https://mlsanity.com/paper/1710.02603/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2017.

## Built on by (most cited first, 8 of 8)

- [DEMix Layers: Disentangling Domains for Modular Language Modeling](https://mlsanity.com/paper/2108.05036/) (2021)
- [From partners to populations: A hierarchical Bayesian account of coordination and convention](https://mlsanity.com/paper/2104.05857/) (2021)
- [Sketch and Customize: A Counterfactual Story Generator](https://mlsanity.com/paper/2104.00929/) (2021)
- [CUE Vectors: Modular Training of Language Models Conditioned on Diverse Contextual Signals](https://mlsanity.com/paper/2203.08774/) (2022)
- [Text Style Transfer: An Introductory Overview](https://mlsanity.com/paper/2407.14822/) (2024)
- [Attention-based Contextual Language Model Adaptation for Speech Recognition](https://mlsanity.com/paper/2106.01451/) (2021)
- [Improving RNN-T ASR Performance with Date-Time and Location Awareness](https://mlsanity.com/paper/2106.06183/) (2021)
- [PriorProbe: Recovering Individual-Level Priors for Personalizing Neural Networks in Facial Expression Recognition](https://mlsanity.com/paper/2602.03882/) (2026)

## Abstract (arXiv)

A context-aware language model uses location, user and/or domain metadata (context) to adapt its predictions. In neural language models, context information is typically represented as an embedding and it is given to the RNN as an additional input, which has been shown to be useful in many applications. We introduce a more powerful mechanism for using context to adapt an RNN by letting the context vector control a low-rank transformation of the recurrent layer weight matrix. Experiments show that allowing a greater fraction of the model parameters to be adjusted has benefits in terms of perplexity and classification for several different types of context.