# Alternative Weighting Schemes for ELMo Embeddings

arXiv: https://arxiv.org/abs/1904.02954 · HTML: https://mlsanity.com/paper/1904.02954/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Apr 2019.

## Built on by (most cited first, 7 of 7)

- [Comparative Analysis Based on DeepSeek, ChatGPT, and Google Gemini: Features, Techniques, Performance, Future Prospects](https://mlsanity.com/paper/2503.04783/) (2025)
- [From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Models](https://mlsanity.com/paper/2411.05036/) (2024)
- [Text classification problems via BERT embedding method and graph convolutional neural network](https://mlsanity.com/paper/2111.15379/) (2021)
- [Computational Approaches to Arabic-English Code-Switching](https://mlsanity.com/paper/2410.13318/) (2024)
- [Investigating Idiomaticity in Word Representations](https://mlsanity.com/paper/2411.02610/) (2024)
- [Evaluating the Efficacy of Length-Controllable Machine Translation](https://mlsanity.com/paper/2305.02300/) (2023)
- [Extracting domain-specific terms using contextual word embeddings](https://mlsanity.com/paper/2502.17278/) (2025)

## Abstract (arXiv)

ELMo embeddings (Peters et. al, 2018) had a huge impact on the NLP community and may recent publications use these embeddings to boost the performance for downstream NLP tasks. However, integration of ELMo embeddings in existent NLP architectures is not straightforward. In contrast to traditional word embeddings, like GloVe or word2vec embeddings, the bi-directional language model of ELMo produces three 1024 dimensional vectors per token in a sentence. Peters et al. proposed to learn a task-specific weighting of these three vectors for downstream tasks. However, this proposed weighting scheme is not feasible for certain tasks, and, as we will show, it does not necessarily yield optimal performance. We evaluate different methods that combine the three vectors from the language model in order to achieve the best possible performance in downstream NLP tasks. We notice that the third layer of the published language model often decreases the performance. By learning a weighted average of only the first two layers, we are able to improve the performance for many datasets. Due to the reduced complexity of the language model, we have a training speed-up of 19-44% for the downstream task.