# Contextual BERT: Conditioning the Language Model Using a Global State

arXiv: https://arxiv.org/abs/2010.15778 · HTML: https://mlsanity.com/paper/2010.15778/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2020.

## Built on by (most cited first, 4 of 4)

- [CUE Vectors: Modular Training of Language Models Conditioned on Diverse Contextual Signals](https://mlsanity.com/paper/2203.08774/) (2022)
- [Smart Fashion: A Review of AI Applications in the Fashion & Apparel Industry](https://mlsanity.com/paper/2111.00905/) (2021)
- [Reusable Self-Attention-based Recommender System for Fashion](https://mlsanity.com/paper/2211.16366/) (2022)
- [KARL-Trans-NER: Knowledge Aware Representation Learning for Named Entity Recognition using Transformers](https://mlsanity.com/paper/2111.15436/) (2021)

## Abstract (arXiv)

BERT is a popular language model whose main pre-training task is to fill in the blank, i.e., predicting a word that was masked out of a sentence, based on the remaining words. In some applications, however, having an additional context can help the model make the right prediction, e.g., by taking the domain or the time of writing into account. This motivates us to advance the BERT architecture by adding a global state for conditioning on a fixed-sized context. We present our two novel approaches and apply them to an industry use-case, where we complete fashion outfits with missing articles, conditioned on a specific customer. An experimental comparison to other methods from the literature shows that our methods improve personalization significantly.