# Contextual Embeddings: When Are They Worth It?

arXiv: https://arxiv.org/abs/2005.09117 · HTML: https://mlsanity.com/paper/2005.09117/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 May 2020.

## Built on by (most cited first, 15 of 15)

- [Decoding EEG Brain Activity for Multi-Modal Natural Language Processing](https://mlsanity.com/paper/2102.08655/) (2021)
- [MicroBERT: Effective Training of Low-resource Monolingual BERTs through Parameter Reduction and Multitask Learning](https://mlsanity.com/paper/2212.12510/) (2022)
- [Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddings](https://mlsanity.com/paper/2210.16848/) (2022)
- [Snowflake: Scaling GNNs to High-Dimensional Continuous Control via Parameter Freezing](https://mlsanity.com/paper/2103.01009/) (2021)
- [Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems Research](https://mlsanity.com/paper/2110.08975/) (2021)
- [Theoretical foundations and limits of word embeddings: what types of meaning can they capture?](https://mlsanity.com/paper/2107.10413/) (2021)
- [Hippocampus: An Efficient and Scalable Memory Module for Agentic AI](https://mlsanity.com/paper/2602.13594/) (2026)
- [Separating Retention from Extraction in the Evaluation of End-to-end Relation Extraction](https://mlsanity.com/paper/2109.12008/) (2021)
- [The Unreasonable Effectiveness of Random Target Embeddings for Continuous-Output Neural Machine Translation](https://mlsanity.com/paper/2310.20620/) (2023)
- [When Embedding Models Meet: Procrustes Bounds and Applications](https://mlsanity.com/paper/2510.13406/) (2025)
- [USE: Dynamic User Modeling with Stateful Sequence Models](https://mlsanity.com/paper/2403.13344/) (2024)
- [Computational Approaches to Arabic-English Code-Switching](https://mlsanity.com/paper/2410.13318/) (2024)
- [Trajectory-Based Meta-Learning for Out-Of-Vocabulary Word Embedding Learning](https://mlsanity.com/paper/2102.12266/) (2021)
- [Semantic maps and metrics for science Semantic maps and metrics for science using deep transformer encoders](https://mlsanity.com/paper/2104.05928/) (2021)
- [Lost in Context? On the Sense-wise Variance of Contextualized Word Embeddings](https://mlsanity.com/paper/2208.09669/) (2022)

## Abstract (arXiv)

We study the settings for which deep contextual embeddings (e.g., BERT) give large improvements in performance relative to classic pretrained embeddings (e.g., GloVe), and an even simpler baseline---random word embeddings---focusing on the impact of the training set size and the linguistic properties of the task. Surprisingly, we find that both of these simpler baselines can match contextual embeddings on industry-scale data, and often perform within 5 to 10% accuracy (absolute) on benchmark tasks. Furthermore, we identify properties of data for which contextual embeddings give particularly large gains: language containing complex structure, ambiguous word usage, and words unseen in training.