# Mimicking Word Embeddings using Subword RNNs

arXiv: https://arxiv.org/abs/1707.06961 · HTML: https://mlsanity.com/paper/1707.06961/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jul 2017.

## Built on by (most cited first, 30 of 30)

- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP](https://mlsanity.com/paper/2112.10508/) (2021)
- [Towards generalisable hate speech detection: a review on obstacles and solutions](https://mlsanity.com/paper/2102.08886/) (2021)
- [Log-based Anomaly Detection Without Log Parsing](https://mlsanity.com/paper/2108.01955/) (2021)
- [What is the Role of Small Models in the LLM Era: A Survey](https://mlsanity.com/paper/2409.06857/) (2024)
- [Zero-Shot Tokenizer Transfer](https://mlsanity.com/paper/2405.07883/) (2024)
- [Dynamic Language Models for Continuously Evolving Content](https://mlsanity.com/paper/2106.06297/) (2021)
- [MINER: Improving Out-of-Vocabulary Named Entity Recognition from an Information Theoretic Perspective](https://mlsanity.com/paper/2204.04391/) (2022)
- [Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Training](https://mlsanity.com/paper/2212.09897/) (2022)
- [Bridging Subword Gaps in Pretrain-Finetune Paradigm for Natural Language Generation](https://mlsanity.com/paper/2106.06125/) (2021)
- [Recent Trends in Named Entity Recognition (NER)](https://mlsanity.com/paper/2101.11420/) (2021)
- [Imputing Out-of-Vocabulary Embeddings with LOVE Makes Language Models Robust with Little Cost](https://mlsanity.com/paper/2203.07860/) (2022)
- [Revisit Out-Of-Vocabulary Problem for Slot Filling: A Unified Contrastive Frameword with Multi-level Data Augmentations](https://mlsanity.com/paper/2302.13584/) (2023)
- [Overcoming Vocabulary Constraints with Pixel-level Fallback](https://mlsanity.com/paper/2504.02122/) (2025)
- [Token Distillation: Attention-aware Input Embeddings For New Tokens](https://mlsanity.com/paper/2505.20133/) (2025)
- [drsphelps at SemEval-2022 Task 2: Learning idiom representations using BERTRAM](https://mlsanity.com/paper/2204.02821/) (2022)
- [Integrating Approaches to Word Representation](https://mlsanity.com/paper/2109.04876/) (2021)
- [Representation Learning via Variational Bayesian Networks](https://mlsanity.com/paper/2306.16326/) (2023)
- [Mitigating Out-of-Entity Errors in Named Entity Recognition: A Sentence-Level Strategy](https://mlsanity.com/paper/2412.08434/) (2024)
- [HYPEROFA: Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initialization](https://mlsanity.com/paper/2504.21018/) (2025)
- [Defragmenting Language Models: An Interpretability-based Approach for Vocabulary Expansion](https://mlsanity.com/paper/2604.16656/) (2026)
- [Learning to Learn to Predict Performance Regressions in Production at Meta](https://mlsanity.com/paper/2208.04351/) (2022)
- [Leveraging knowledge graphs to update scientific word embeddings using latent semantic imputation](https://mlsanity.com/paper/2210.15358/) (2022)
- [TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment](https://mlsanity.com/paper/2605.13429/) (2026)
- [Trajectory-Based Meta-Learning for Out-Of-Vocabulary Word Embedding Learning](https://mlsanity.com/paper/2102.12266/) (2021)
- [API-Miner: an API-to-API Specification Recommendation Engine](https://mlsanity.com/paper/2212.07253/) (2022)
- [Advancements in eHealth Data Analytics through Natural Language Processing and Deep Learning](https://mlsanity.com/paper/2401.10850/) (2024)
- [Reconsidering Degeneration of Token Embeddings with Definitions for Encoder-based Pre-trained Language Models](https://mlsanity.com/paper/2408.01308/) (2024)
- [Handling Korean Out-of-Vocabulary Words with Phoneme Representation Learning](https://mlsanity.com/paper/2507.04018/) (2025)
- [DN-Hypo-Pipeline: An AI-Driven Workflow for Generating Hypotheses using Large Language Models and Scientific Explanations](https://mlsanity.com/paper/2606.08532/) (2026)

## Abstract (arXiv)

Word embeddings improve generalization over lexical features by placing each word in a lower-dimensional space, using distributional information obtained from unlabeled data. However, the effectiveness of word embeddings for downstream NLP tasks is limited by out-of-vocabulary (OOV) words, for which embeddings do not exist. In this paper, we present MIMICK, an approach to generating OOV word embeddings compositionally, by learning a function from spellings to distributional embeddings. Unlike prior work, MIMICK does not require re-training on the original word embedding corpus; instead, learning is performed at the type level. Intrinsic and extrinsic evaluations demonstrate the power of this simple approach. On 23 languages, MIMICK improves performance over a word-based baseline for tagging part-of-speech and morphosyntactic attributes. It is competitive with (and complementary to) a supervised character-based model in low-resource settings.