# Dynamic Meta-Embeddings for Improved Sentence Representations

arXiv: https://arxiv.org/abs/1804.07983 · HTML: https://mlsanity.com/paper/1804.07983/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Apr 2018.

## Built on by (most cited first, 15 of 15)

- [A General Survey on Attention Mechanisms in Deep Learning](https://mlsanity.com/paper/2203.14263/) (2022)
- [Are Multilingual Models Effective in Code-Switching?](https://mlsanity.com/paper/2103.13309/) (2021)
- [How Transferable Are Self-supervised Features in Medical Image Classification Tasks?](https://mlsanity.com/paper/2108.10048/) (2021)
- [Attentive Neural Controlled Differential Equations for Time-series Classification and Forecasting](https://mlsanity.com/paper/2109.01876/) (2021)
- [ACE-NODE: Attentive Co-Evolving Neural Ordinary Differential Equations](https://mlsanity.com/paper/2105.14953/) (2021)
- [Learning Graph Meta Embeddings for Cold-Start Ads in Click-Through Rate Prediction](https://mlsanity.com/paper/2105.08909/) (2021)
- [A Survey on Word Meta-Embedding Learning](https://mlsanity.com/paper/2204.11660/) (2022)
- [TADA: Efficient Task-Agnostic Domain Adaptation for Transformers](https://mlsanity.com/paper/2305.12717/) (2023)
- [Learning Meta Word Embeddings by Unsupervised Weighted Concatenation of Source Embeddings](https://mlsanity.com/paper/2204.12386/) (2022)
- [Unsupervised Attention-based Sentence-Level Meta-Embeddings from Contextualised Language Models](https://mlsanity.com/paper/2204.07746/) (2022)
- [A Meta-embedding-based Ensemble Approach for ICD Coding Prediction](https://mlsanity.com/paper/2102.13622/) (2021)
- [Denoising Word Embeddings by Averaging in a Shared Space](https://mlsanity.com/paper/2106.02954/) (2021)
- [Looking to Learn: Token-wise Dynamic Gating for Low-Resource Vision-Language Modelling](https://mlsanity.com/paper/2510.08470/) (2025)
- [Together We Make Sense -- Learning Meta-Sense Embeddings from Pretrained Static Sense Embeddings](https://mlsanity.com/paper/2305.19092/) (2023)
- [Dual-Query Multiple Instance Learning for Dynamic Meta-Embedding based Tumor Classification](https://mlsanity.com/paper/2307.07482/) (2023)

## Abstract (arXiv)

While one of the first steps in many NLP systems is selecting what pre-trained word embeddings to use, we argue that such a step is better left for neural networks to figure out by themselves. To that end, we introduce dynamic meta-embeddings, a simple yet effective method for the supervised learning of embedding ensembles, which leads to state-of-the-art performance within the same model class on a variety of tasks. We subsequently show how the technique can be used to shed new light on the usage of word embeddings in NLP systems.