# Retrofitting Contextualized Word Embeddings with Paraphrases

arXiv: https://arxiv.org/abs/1909.09700 · HTML: https://mlsanity.com/paper/1909.09700/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Sep 2019.

## Built on by (most cited first, 6 of 6)

- [ACCoRD: A Multi-Document Approach to Generating Diverse Descriptions of Scientific Concepts](https://mlsanity.com/paper/2205.06982/) (2022)
- [Learning Sense-Specific Static Embeddings using Contextualised Word Embeddings as a Proxy](https://mlsanity.com/paper/2110.02204/) (2021)
- [Evaluation of taxonomic and neural embedding methods for calculating semantic similarity](https://mlsanity.com/paper/2209.15197/) (2022)
- [Retrofitting Light-weight Language Models for Emotions using Supervised Contrastive Learning](https://mlsanity.com/paper/2310.18930/) (2023)
- [Lost in Context? On the Sense-wise Variance of Contextualized Word Embeddings](https://mlsanity.com/paper/2208.09669/) (2022)
- [Distilling Monolingual and Crosslingual Word-in-Context Representations](https://mlsanity.com/paper/2409.08719/) (2024)

## Abstract (arXiv)

Contextualized word embedding models, such as ELMo, generate meaningful representations of words and their context. These models have been shown to have a great impact on downstream applications. However, in many cases, the contextualized embedding of a word changes drastically when the context is paraphrased. As a result, the downstream model is not robust to paraphrasing and other linguistic variations. To enhance the stability of contextualized word embedding models, we propose an approach to retrofitting contextualized embedding models with paraphrase contexts. Our method learns an orthogonal transformation on the input space, which seeks to minimize the variance of word representations on paraphrased contexts. Experiments show that the retrofitted model significantly outperforms the original ELMo on various sentence classification and language inference tasks.