# CUE Vectors: Modular Training of Language Models Conditioned on Diverse Contextual Signals

arXiv: https://arxiv.org/abs/2203.08774 · HTML: https://mlsanity.com/paper/2203.08774/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 16 Mar 2022.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates](https://mlsanity.com/paper/1708.07120/) (2017)
- [Contextual RNN-T For Open Domain ASR](https://mlsanity.com/paper/2006.03411/) (2020)
- [Improving the Transformer Translation Model with Document-Level Context](https://mlsanity.com/paper/1810.03581/) (2018)
- [Dialog-context aware end-to-end speech recognition](https://mlsanity.com/paper/1808.02171/) (2018)
- [Personalized Language Model for Query Auto-Completion](https://mlsanity.com/paper/1804.09661/) (2018)
- [Fast Contextual Adaptation with Neural Associative Memory for On-Device Personalized Speech Recognition](https://mlsanity.com/paper/2110.02220/) (2021)
- [Larger-Context Language Modelling](https://mlsanity.com/paper/1511.03729/) (2015)
- [Document Context Language Models](https://mlsanity.com/paper/1511.03962/) (2015)
- [Personalized neural language models for real-world query auto completion](https://mlsanity.com/paper/1804.06439/) (2018)
- [Low-Rank RNN Adaptation for Context-Aware Language Modeling](https://mlsanity.com/paper/1710.02603/) (2017)
- [Dialog Context Language Modeling with Recurrent Neural Networks](https://mlsanity.com/paper/1701.04056/) (2017)
- [Context-aware Decoder for Neural Machine Translation using a Target-side Document-Level Language Model](https://mlsanity.com/paper/2010.12827/) (2020)
- [Contextual BERT: Conditioning the Language Model Using a Global State](https://mlsanity.com/paper/2010.15778/) (2020)
- [Contextual Biasing of Language Models for Speech Recognition in Goal-Oriented Conversational Agents](https://mlsanity.com/paper/2103.10325/) (2021)
- [Attention-based Contextual Language Model Adaptation for Speech Recognition](https://mlsanity.com/paper/2106.01451/) (2021)
- [Hierarchical GPT with Congruent Transformers for Multi-Sentence Language Models](https://mlsanity.com/paper/2009.08636/) (2020)

## Built on by (most cited first, 5 of 5)

- [Using External Off-Policy Speech-To-Text Mappings in Contextual End-To-End Automated Speech Recognition](https://mlsanity.com/paper/2301.02736/) (2023)
- [Reference-less Analysis of Context Specificity in Translation with Personalised Language Models](https://mlsanity.com/paper/2303.16618/) (2023)
- [MTCue: Learning Zero-Shot Control of Extra-Textual Attributes by Leveraging Unstructured Context in Neural Machine Translation](https://mlsanity.com/paper/2305.15904/) (2023)
- [An Efficient Self-Learning Framework For Interactive Spoken Dialog Systems](https://mlsanity.com/paper/2409.10515/) (2024)
- [LIME: Making LLM Data More Efficient with Linguistic Metadata Embeddings](https://mlsanity.com/paper/2512.07522/) (2025)

## Abstract (arXiv)

We propose a framework to modularize the training of neural language models that use diverse forms of sentence-external context (including metadata) by eliminating the need to jointly train sentence-external and within-sentence encoders. Our approach, contextual universal embeddings (CUE), trains LMs on one set of context, such as date and author, and adapts to novel metadata types, such as article title, or previous sentence. The model consists of a pretrained neural sentence LM, a BERT-based context encoder, and a masked transformer decoder that estimates LM probabilities using sentence-internal and sentence-external information. When context or metadata are unavailable, our model learns to combine contextual and sentence-internal information using noisy oracle unigram embeddings as a proxy. Real contextual information can be introduced later and used to adapt a small number of parameters that map contextual data into the decoder's embedding space. We validate the CUE framework on a NYTimes text corpus with multiple metadata types, for which the LM perplexity can be lowered from 36.6 to 27.4 by conditioning on context. Bootstrapping a contextual LM with only a subset of the context/metadata during training retains 85% of the achievable gain. Training the model initially with proxy context retains 67% of the perplexity gain after adapting to real context. Furthermore, we can swap one type of pretrained sentence LM for another without retraining the context encoders, by only adapting the decoder model. Overall, we obtain a modular framework that allows incremental, scalable training of context-enhanced LMs.