# Few-Shot Representation Learning for Out-Of-Vocabulary Words

arXiv: https://arxiv.org/abs/1907.00505 · HTML: https://mlsanity.com/paper/1907.00505/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jul 2019.

## Built on by (most cited first, 30 of 30)

- [Log-based Anomaly Detection Without Log Parsing](https://mlsanity.com/paper/2108.01955/) (2021)
- [Dynamic Language Models for Continuously Evolving Content](https://mlsanity.com/paper/2106.06297/) (2021)
- [Meta Learning for Natural Language Processing: A Survey](https://mlsanity.com/paper/2205.01500/) (2022)
- [NEO-BENCH: Evaluating Robustness of Large Language Models with Neologisms](https://mlsanity.com/paper/2402.12261/) (2024)
- [MINER: Improving Out-of-Vocabulary Named Entity Recognition from an Information Theoretic Perspective](https://mlsanity.com/paper/2204.04391/) (2022)
- [Meta-Inductive Node Classification across Graphs](https://mlsanity.com/paper/2105.06725/) (2021)
- [ReFactor GNNs: Revisiting Factorisation-based Models from a Message-Passing Perspective](https://mlsanity.com/paper/2207.09980/) (2022)
- [Few-Shot Learning on Graphs](https://mlsanity.com/paper/2203.09308/) (2022)
- [What Can Knowledge Bring to Machine Learning? -- A Survey of Low-shot Learning for Structured Data](https://mlsanity.com/paper/2106.06410/) (2021)
- [Imputing Out-of-Vocabulary Embeddings with LOVE Makes Language Models Robust with Little Cost](https://mlsanity.com/paper/2203.07860/) (2022)
- [BinBert: Binary Code Understanding with a Fine-tunable and Execution-aware Transformer](https://mlsanity.com/paper/2208.06692/) (2022)
- [LOSTIN: Logic Optimization via Spatio-Temporal Information with Hybrid Graph Models](https://mlsanity.com/paper/2201.08455/) (2022)
- [A Survey of Ontology Expansion for Conversational Understanding](https://mlsanity.com/paper/2410.15019/) (2024)
- [A Multi-Strategy based Pre-Training Method for Cold-Start Recommendation](https://mlsanity.com/paper/2112.02275/) (2021)
- [Lacking the embedding of a word? Look it up into a traditional dictionary](https://mlsanity.com/paper/2109.11763/) (2021)
- [Rapid Word Learning Through Meta In-Context Learning](https://mlsanity.com/paper/2502.14791/) (2025)
- [Representation Learning via Variational Bayesian Networks](https://mlsanity.com/paper/2306.16326/) (2023)
- [CoLLEGe: Concept Embedding Generation for Large Language Models](https://mlsanity.com/paper/2403.15362/) (2024)
- [Mitigating Out-of-Entity Errors in Named Entity Recognition: A Sentence-Level Strategy](https://mlsanity.com/paper/2412.08434/) (2024)
- [Novel Slot Detection: A Benchmark for Discovering Unknown Slot Types in the Task-Oriented Dialogue System](https://mlsanity.com/paper/2105.14313/) (2021)
- [Unveiling Project-Specific Bias in Neural Code Models](https://mlsanity.com/paper/2201.07381/) (2022)
- [TransDrift: Modeling Word-Embedding Drift using Transformer](https://mlsanity.com/paper/2206.08081/) (2022)
- [Characterizing and Measuring Linguistic Dataset Drift](https://mlsanity.com/paper/2305.17127/) (2023)
- [Trajectory-Based Meta-Learning for Out-Of-Vocabulary Word Embedding Learning](https://mlsanity.com/paper/2102.12266/) (2021)
- [Self-supervised Graph Learning for Occasional Group Recommendation](https://mlsanity.com/paper/2112.02274/) (2021)
- [Learning node embeddings via summary graphs: a brief theoretical analysis](https://mlsanity.com/paper/2207.01189/) (2022)
- [DeviceBERT: Applied Transfer Learning With Targeted Annotations and Vocabulary Enrichment to Identify Medical Device and Component Terminology in FDA Recall Summaries](https://mlsanity.com/paper/2406.05307/) (2024)
- [A Comprehensive Survey of Challenges and Opportunities of Few-Shot Learning Across Multiple Domains](https://mlsanity.com/paper/2504.04017/) (2025)
- [Handling Korean Out-of-Vocabulary Words with Phoneme Representation Learning](https://mlsanity.com/paper/2507.04018/) (2025)
- [Evaluating Embedding Frameworks for Scientific Domain](https://mlsanity.com/paper/2510.06244/) (2025)

## Abstract (arXiv)

Existing approaches for learning word embeddings often assume there are sufficient occurrences for each word in the corpus, such that the representation of words can be accurately estimated from their contexts. However, in real-world scenarios, out-of-vocabulary (a.k.a. OOV) words that do not appear in training corpus emerge frequently. It is challenging to learn accurate representations of these words with only a few observations. In this paper, we formulate the learning of OOV embeddings as a few-shot regression problem, and address it by training a representation function to predict the oracle embedding vector (defined as embedding trained with abundant observations) based on limited observations. Specifically, we propose a novel hierarchical attention-based architecture to serve as the neural regression function, with which the context information of a word is encoded and aggregated from K observations. Furthermore, our approach can leverage Model-Agnostic Meta-Learning (MAML) for adapting the learned model to the new corpus fast and robustly. Experiments show that the proposed approach significantly outperforms existing methods in constructing accurate embeddings for OOV words, and improves downstream tasks where these embeddings are utilized.