# BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance

arXiv: https://arxiv.org/abs/1910.07181 · HTML: https://mlsanity.com/paper/1910.07181/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2019.

## Built on by (most cited first, 19 of 19)

- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding](https://mlsanity.com/paper/2204.10050/) (2022)
- [Zero-Shot Tokenizer Transfer](https://mlsanity.com/paper/2405.07883/) (2024)
- [Searching for Optimal Subword Tokenization in Cross-domain NER](https://mlsanity.com/paper/2206.03352/) (2022)
- [Retrofitting Large Language Models with Dynamic Tokenization](https://mlsanity.com/paper/2411.18553/) (2024)
- [MDAPT: Multilingual Domain Adaptive Pretraining in a Single Model](https://mlsanity.com/paper/2109.06605/) (2021)
- [The Impact of Word Splitting on the Semantic Content of Contextualized Word Representations](https://mlsanity.com/paper/2402.14616/) (2024)
- [Token Distillation: Attention-aware Input Embeddings For New Tokens](https://mlsanity.com/paper/2505.20133/) (2025)
- [drsphelps at SemEval-2022 Task 2: Learning idiom representations using BERTRAM](https://mlsanity.com/paper/2204.02821/) (2022)
- [Lacking the embedding of a word? Look it up into a traditional dictionary](https://mlsanity.com/paper/2109.11763/) (2021)
- [Sample Efficient Approaches for Idiomaticity Detection](https://mlsanity.com/paper/2205.11306/) (2022)
- [HYPEROFA: Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initialization](https://mlsanity.com/paper/2504.21018/) (2025)
- [Overcoming Poor Word Embeddings with Word Definitions](https://mlsanity.com/paper/2103.03842/) (2021)
- [Probing Commonsense Knowledge in Pre-trained Language Models with Sense-level Precision and Expanded Vocabulary](https://mlsanity.com/paper/2210.06376/) (2022)
- [READIN: A Chinese Multi-Task Benchmark with Realistic and Diverse Input Noises](https://mlsanity.com/paper/2302.07324/) (2023)
- [Towards Efficient Methods in Medical Question Answering using Knowledge Graph Embeddings](https://mlsanity.com/paper/2401.07977/) (2024)
- [TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment](https://mlsanity.com/paper/2605.13429/) (2026)
- [Unsupervised Paraphrasing of Multiword Expressions](https://mlsanity.com/paper/2306.01443/) (2023)
- [SMI: Ontology-Grounded Semantic IDs for Hierarchy-Aware Medical Concept Representation](https://mlsanity.com/paper/2610.18672/) (2026)

## Abstract (arXiv)

Pretraining deep language models has led to large performance gains in NLP. Despite this success, Schick and Sch"utze (2020) recently showed that these models struggle to understand rare words. For static word embeddings, this problem has been addressed by separately learning representations for rare words. In this work, we transfer this idea to pretrained language models: We introduce BERTRAM, a powerful architecture based on BERT that is capable of inferring high-quality embeddings for rare words that are suitable as input representations for deep language models. This is achieved by enabling the surface form and contexts of a word to interact with each other in a deep architecture. Integrating BERTRAM into BERT leads to large performance increases due to improved representations of rare and medium frequency words on both a rare word probing task and three downstream tasks.