# Multiple Word Embeddings for Increased Diversity of Representation

arXiv: https://arxiv.org/abs/2009.14394 · HTML: https://mlsanity.com/paper/2009.14394/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Sep 2020.

## Built on by (most cited first, 4 of 4)

- [The Art of Embedding Fusion: Optimizing Hate Speech Detection](https://mlsanity.com/paper/2306.14939/) (2023)
- [Intent Features for Rich Natural Language Understanding](https://mlsanity.com/paper/2104.08701/) (2021)
- [CEFER: A Four Facets Framework based on Context and Emotion embedded features for Implicit and Explicit Emotion Recognition](https://mlsanity.com/paper/2209.13999/) (2022)
- [TriagerX: Dual Transformers for Bug Triaging Tasks with Content and Interaction Based Rankings](https://mlsanity.com/paper/2508.16860/) (2025)

## Abstract (arXiv)

Most state-of-the-art models in natural language processing (NLP) are neural models built on top of large, pre-trained, contextual language models that generate representations of words in context and are fine-tuned for the task at hand. The improvements afforded by these "contextual embeddings" come with a high computational cost. In this work, we explore a simple technique that substantially and consistently improves performance over a strong baseline with negligible increase in run time. We concatenate multiple pre-trained embeddings to strengthen our representation of words. We show that this concatenation technique works across many tasks, datasets, and model types. We analyze aspects of pre-trained embedding similarity and vocabulary coverage and find that the representational diversity between different pre-trained embeddings is the driving force of why this technique works. We provide open source implementations of our models in both TensorFlow and PyTorch.