# Predicting and interpreting embeddings for out of vocabulary words in downstream tasks

arXiv: https://arxiv.org/abs/1903.00724 · HTML: https://mlsanity.com/paper/1903.00724/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Mar 2019.

## Built on by (most cited first, 5 of 5)

- [NEO-BENCH: Evaluating Robustness of Large Language Models with Neologisms](https://mlsanity.com/paper/2402.12261/) (2024)
- [Trajectory-Based Meta-Learning for Out-Of-Vocabulary Word Embedding Learning](https://mlsanity.com/paper/2102.12266/) (2021)
- [EasyNER: A Customizable Easy-to-Use Pipeline for Deep Learning- and Dictionary-based Named Entity Recognition from Medical and Life Science Text](https://mlsanity.com/paper/2304.07805/) (2023)
- [DeviceBERT: Applied Transfer Learning With Targeted Annotations and Vocabulary Enrichment to Identify Medical Device and Component Terminology in FDA Recall Summaries](https://mlsanity.com/paper/2406.05307/) (2024)
- [KoNeoBench: A Curated Evaluation Dataset for LLM Understanding of Korean Neologisms](https://mlsanity.com/paper/2609.19916/) (2026)

## Abstract (arXiv)

We propose a novel way to handle out of vocabulary (OOV) words in downstream natural language processing (NLP) tasks. We implement a network that predicts useful embeddings for OOV words based on their morphology and on the context in which they appear. Our model also incorporates an attention mechanism indicating the focus allocated to the left context words, the right context words or the word's characters, hence making the prediction more interpretable. The model is a ``drop-in'' module that is jointly trained with the downstream task's neural network, thus producing embeddings specialized for the task at hand. When the task is mostly syntactical, we observe that our model aims most of its attention on surface form characters. On the other hand, for tasks more semantical, the network allocates more attention to the surrounding words. In all our tests, the module helps the network to achieve better performances in comparison to the use of simple random embeddings.