# Robust Embeddings Via Distributions

arXiv: https://arxiv.org/abs/2104.08420 · HTML: https://mlsanity.com/paper/2104.08420/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 26 references that are in the map.
- First posted on arXiv 17 Apr 2021.

## Builds on (most cited first, 15 of 15)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Quantifying the Carbon Emissions of Machine Learning](https://mlsanity.com/paper/1910.09700/) (2019)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [Word Representations via Gaussian Embedding](https://mlsanity.com/paper/1412.6623/) (2014)
- [On the Impact of Various Types of Noise on Neural Machine Translation](https://mlsanity.com/paper/1805.12282/) (2018)
- [Misspelling Oblivious Word Embeddings](https://mlsanity.com/paper/1905.09755/) (2019)
- [Contextual Text Denoising with Masked Language Models](https://mlsanity.com/paper/1910.14080/) (2019)
- [Sentence Embeddings in NLI with Iterative Refinement Encoders](https://mlsanity.com/paper/1808.08762/) (2018)
- [Gaussian Word Embedding with a Wasserstein Distance Loss](https://mlsanity.com/paper/1808.07016/) (2018)
- [Towards robust word embeddings for noisy texts](https://mlsanity.com/paper/1911.10876/) (2019)
- [Kernelized Bayesian Softmax for Text Generation](https://mlsanity.com/paper/1911.00274/) (2019)

## Built on by (most cited first, 1 of 1)

- [A Fast Randomized Algorithm for Massive Text Normalization](https://mlsanity.com/paper/2110.03024/) (2021)

## Abstract (arXiv)

Despite recent monumental advances in the field, many Natural Language Processing (NLP) models still struggle to perform adequately on noisy domains. We propose a novel probabilistic embedding-level method to improve the robustness of NLP models. Our method, Robust Embeddings via Distributions (RED), incorporates information from both noisy tokens and surrounding context to obtain distributions over embedding vectors that can express uncertainty in semantic space more fully than any deterministic method. We evaluate our method on a number of downstream tasks using existing state-of-the-art models in the presence of both natural and synthetic noise, and demonstrate a clear improvement over other embedding approaches to robustness from the literature.