# Rare Words: A Major Problem for Contextualized Embeddings And How to Fix it by Attentive Mimicking

arXiv: https://arxiv.org/abs/1904.06707 · HTML: https://mlsanity.com/paper/1904.06707/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Apr 2019.

## Built on by (most cited first, 30 of 41)

- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [Unmasking the Mask -- Evaluating Social Biases in Masked Language Models](https://mlsanity.com/paper/2104.07496/) (2021)
- [Dict-BERT: Enhancing Language Model Pre-training with Dictionary](https://mlsanity.com/paper/2110.06490/) (2021)
- [Greed is All You Need: An Evaluation of Tokenizer Inference Methods](https://mlsanity.com/paper/2403.01289/) (2024)
- [Towards Open-World Feature Extrapolation: An Inductive Graph Learning Approach](https://mlsanity.com/paper/2110.04514/) (2021)
- [AVocaDo: Strategy for Adapting Vocabulary to Downstream Domain](https://mlsanity.com/paper/2110.13434/) (2021)
- [Improving Tokenisation by Alternative Treatment of Spaces](https://mlsanity.com/paper/2204.04058/) (2022)
- [WinoDict: Probing language models for in-context word acquisition](https://mlsanity.com/paper/2209.12153/) (2022)
- [Always Keep your Target in Mind: Studying Semantics and Improving Performance of Neural Lexical Substitution](https://mlsanity.com/paper/2206.11815/) (2022)
- [AnnoBERT: Effectively Representing Multiple Annotators' Label Choices to Improve Hate Speech Detection](https://mlsanity.com/paper/2212.10405/) (2022)
- [Can Character-based Language Models Improve Downstream Task Performance in Low-Resource and Noisy Language Scenarios?](https://mlsanity.com/paper/2110.13658/) (2021)
- [Leveraging Visual Knowledge in Language Tasks: An Empirical Study on Intermediate Pre-training for Cross-modal Knowledge Transfer](https://mlsanity.com/paper/2203.07519/) (2022)
- [WSPAlign: Word Alignment Pre-training via Large-Scale Weakly Supervised Span Prediction](https://mlsanity.com/paper/2306.05644/) (2023)
- [Imputing Out-of-Vocabulary Embeddings with LOVE Makes Language Models Robust with Little Cost](https://mlsanity.com/paper/2203.07860/) (2022)
- [MEDVOC: Vocabulary Adaptation for Fine-tuning Pre-trained Language Models on Medical Text Summarization](https://mlsanity.com/paper/2405.04163/) (2024)
- [A Comparison of SVM against Pre-trained Language Models (PLMs) for Text Classification Tasks](https://mlsanity.com/paper/2211.02563/) (2022)
- [Extracting Sentence Embeddings from Pretrained Transformer Models](https://mlsanity.com/paper/2408.08073/) (2024)
- [Managing ML Pipelines: Feature Stores and the Coming Wave of Embedding Ecosystems](https://mlsanity.com/paper/2108.05053/) (2021)
- [Prompt Combines Paraphrase: Teaching Pre-trained Models to Understand Rare Biomedical Words](https://mlsanity.com/paper/2209.06453/) (2022)
- [The Impact of Word Splitting on the Semantic Content of Contextualized Word Representations](https://mlsanity.com/paper/2402.14616/) (2024)
- [Explicit Morphological Knowledge Improves Pre-training of Language Models for Hebrew](https://mlsanity.com/paper/2311.00658/) (2023)
- [Does He Wink or Does He Nod? A Challenging Benchmark for Evaluating Word Understanding of Language Models](https://mlsanity.com/paper/2102.03596/) (2021)
- [Artefact Retrieval: Overview of NLP Models with Knowledge Base Access](https://mlsanity.com/paper/2201.09651/) (2022)
- [The KITMUS Test: Evaluating Knowledge Integration from Multiple Sources in Natural Language Understanding Systems](https://mlsanity.com/paper/2212.08192/) (2022)
- [Rapid Word Learning Through Meta In-Context Learning](https://mlsanity.com/paper/2502.14791/) (2025)
- [Combining Context-Free and Contextualized Representations for Arabic Sarcasm Detection and Sentiment Identification](https://mlsanity.com/paper/2103.05683/) (2021)
- [Generating Coherent Drum Accompaniment With Fills And Improvisations](https://mlsanity.com/paper/2209.00291/) (2022)
- [Made of Steel? Learning Plausible Materials for Components in the Vehicle Repair Domain](https://mlsanity.com/paper/2304.14745/) (2023)

## Abstract (arXiv)

Pretraining deep neural network architectures with a language modeling objective has brought large improvements for many natural language processing tasks. Exemplified by BERT, a recently proposed such architecture, we demonstrate that despite being trained on huge amounts of data, deep language models still struggle to understand rare words. To fix this problem, we adapt Attentive Mimicking, a method that was designed to explicitly learn embeddings for rare words, to deep language models. In order to make this possible, we introduce one-token approximation, a procedure that enables us to use Attentive Mimicking even when the underlying language model uses subword-based tokenization, i.e., it does not assign embeddings to all words. To evaluate our method, we create a novel dataset that tests the ability of language models to capture semantic properties of words without any task-specific fine-tuning. Using this dataset, we show that adding our adapted version of Attentive Mimicking to BERT does indeed substantially improve its understanding of rare words.