Contextual Embeddings: When Are They Worth It?
Not on the map. Why?
Abstract (arXiv)
We study the settings for which deep contextual embeddings (e.g., BERT) give large improvements in performance relative to classic pretrained embeddings (e.g., GloVe), and an even simpler baseline---random word embeddings---focusing on the impact of the training set size and the linguistic properties of the task. Surprisingly, we find that both of these simpler baselines can match contextual embeddings on industry-scale data, and often perform within 5 to 10% accuracy (absolute) on benchmark tasks. Furthermore, we identify properties of data for which contextual embeddings give particularly large gains: language containing complex structure, ambiguous word usage, and words unseen in training.
Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 May 2020.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 7; 2022: 3; 2023: 1; 2024: 2; 2025: 1; 2026: 1.
15 papers.
Sort:
By area:
- 2021Decoding EEG Brain Activity for Multi-Modal Natural Language Processingcited by 13
- 2022MicroBERT: Effective Training of Low-resource Monolingual BERTs through Parameter Reduction and Multitask Learningcited by 12
- 2022Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddingscited by 11
- 2021Snowflake: Scaling GNNs to High-Dimensional Continuous Control via Parameter Freezingcited by 7
- 2021Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems Researchcited by 7
- 2021Theoretical foundations and limits of word embeddings: what types of meaning can they capture?cited by 5
- 2026Hippocampus: An Efficient and Scalable Memory Module for Agentic AIcited by 3
- 2021Separating Retention from Extraction in the Evaluation of End-to-end Relation Extractioncited by 2
- 2023The Unreasonable Effectiveness of Random Target Embeddings for Continuous-Output Neural Machine Translationcited by 2
- 2024USE: Dynamic User Modeling with Stateful Sequence Modelscited by 1
- 2024Computational Approaches to Arabic-English Code-Switchingcited by 1
- 2025When Embedding Models Meet: Procrustes Bounds and Applicationscited by 1
3 more
- 2021Trajectory-Based Meta-Learning for Out-Of-Vocabulary Word Embedding Learningcited by 0
- 2021Semantic maps and metrics for science Semantic maps and metrics for science using deep transformer encoderscited by 0
- 2022Lost in Context? On the Sense-wise Variance of Contextualized Word Embeddingscited by 0
- 2026Hippocampus: An Efficient and Scalable Memory Module for Agentic AI14 Feb 2026
- 2025When Embedding Models Meet: Procrustes Bounds and Applications15 Oct 2025
- 2024Computational Approaches to Arabic-English Code-Switching17 Oct 2024
- 2024USE: Dynamic User Modeling with Stateful Sequence Models20 Mar 2024
- 2023The Unreasonable Effectiveness of Random Target Embeddings for Continuous-Output Neural Machine Translation31 Oct 2023
- 2022MicroBERT: Effective Training of Low-resource Monolingual BERTs through Parameter Reduction and Multitask Learning23 Dec 2022
- 2022Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddings30 Oct 2022
- 2022Lost in Context? On the Sense-wise Variance of Contextualized Word Embeddings20 Aug 2022
- 2021Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems Research18 Oct 2021
- 2021Separating Retention from Extraction in the Evaluation of End-to-end Relation Extraction24 Sep 2021
- 2021Theoretical foundations and limits of word embeddings: what types of meaning can they capture?22 Jul 2021
- 2021Semantic maps and metrics for science Semantic maps and metrics for science using deep transformer encoders13 Apr 2021
- 2021Decoding EEG Brain Activity for Multi-Modal Natural Language Processingcited by 13
- 2022MicroBERT: Effective Training of Low-resource Monolingual BERTs through Parameter Reduction and Multitask Learningcited by 12
- 2022Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddingscited by 11
- 2021Snowflake: Scaling GNNs to High-Dimensional Continuous Control via Parameter Freezingcited by 7
- 2021Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems Researchcited by 7
- 2021Theoretical foundations and limits of word embeddings: what types of meaning can they capture?cited by 5
- 2021Separating Retention from Extraction in the Evaluation of End-to-end Relation Extractioncited by 2
- 2023The Unreasonable Effectiveness of Random Target Embeddings for Continuous-Output Neural Machine Translationcited by 2
- 2024USE: Dynamic User Modeling with Stateful Sequence Modelscited by 1
- 2024Computational Approaches to Arabic-English Code-Switchingcited by 1
- 2025When Embedding Models Meet: Procrustes Bounds and Applicationscited by 1
- 2021Trajectory-Based Meta-Learning for Out-Of-Vocabulary Word Embedding Learningcited by 0
12 most cited of 14 All 14 (JSON)
- 2025When Embedding Models Meet: Procrustes Bounds and Applications15 Oct 2025
- 2024Computational Approaches to Arabic-English Code-Switching17 Oct 2024
- 2024USE: Dynamic User Modeling with Stateful Sequence Models20 Mar 2024
- 2023The Unreasonable Effectiveness of Random Target Embeddings for Continuous-Output Neural Machine Translation31 Oct 2023
- 2022MicroBERT: Effective Training of Low-resource Monolingual BERTs through Parameter Reduction and Multitask Learning23 Dec 2022
- 2022Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddings30 Oct 2022
- 2022Lost in Context? On the Sense-wise Variance of Contextualized Word Embeddings20 Aug 2022
- 2021Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems Research18 Oct 2021
- 2021Separating Retention from Extraction in the Evaluation of End-to-end Relation Extraction24 Sep 2021
- 2021Theoretical foundations and limits of word embeddings: what types of meaning can they capture?22 Jul 2021
- 2021Semantic maps and metrics for science Semantic maps and metrics for science using deep transformer encoders13 Apr 2021
- 2021Snowflake: Scaling GNNs to High-Dimensional Continuous Control via Parameter Freezing1 Mar 2021
12 newest of 14 All 14 (JSON)