mlsanityData to 7 Oct 2026

A Survey on Contextual Embeddings

arXiv 2003.07278 ↗First posted 16 Mar 2020cs.CL cs.AI cs.LGcited by 48

Not on the map. Why?

Abstract (arXiv)

Contextual embeddings, such as ELMo and BERT, move beyond global word representations like Word2Vec and achieve ground-breaking performance on a wide range of natural language processing tasks. Contextual embeddings assign each word a representation based on its context, thereby capturing uses of words across varied contexts and encoding knowledge that transfers across languages. In this survey, we review existing contextual embedding models, cross-lingual polyglot pre-training, the application of contextual embeddings in downstream tasks, model compression, and model analyses.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 15; 2022: 6; 2023: 15; 2024: 6; 2025: 6.

48 papers.

Sort:

  1. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  2. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  3. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  4. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  5. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  6. 2023Scaling laws for language encoding models in fMRIcited by 68
  7. 2022Multimodal learning with graphscited by 54
  8. 2021AMMU : A Survey of Transformer-based Biomedical Pretrained Language Modelscited by 32
  9. 2022Findings of the The RuATD Shared Task 2022 on Artificial Text Detection in Russiancited by 29
  10. 2021Are Pre-trained Convolutions Better than Pre-trained Transformers?cited by 27
  11. 2023A Family of Pretrained Transformer Language Models for Russiancited by 27
  12. 2021On Cross-Lingual Retrieval with Multilingual Text Encoderscited by 22
36 more
  1. 2024Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmarkcited by 19
  2. 2024Understanding LLM Embeddings for Regressioncited by 19
  3. 2023Towards Automatic Boundary Detection for Human-AI Collaborative Hybrid Essay in Educationcited by 17
  4. 2021Evaluating Multilingual Text Encoders for Unsupervised Cross-Lingual Retrievalcited by 16
  5. 2023Linguistic ambiguity analysis in ChatGPTcited by 14
  6. 2023Pre-trained Embeddings for Entity Resolution: An Experimental Analysis [Experiment, Analysis & Benchmark]cited by 13
  7. 2025Anatomy of a Machine Learning Ecosystem: 2 Million Models on Hugging Facecited by 12
  8. 2021Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbationscited by 10
  9. 2024Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Surveycited by 10
  10. 2023Soft-prompt Tuning for Large Language Models to Evaluate Biascited by 6
  11. 2023Can Instruction Fine-Tuned Language Models Identify Social Bias through Prompting?cited by 6
  12. 2024From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Modelscited by 6
  13. 2023Learning Defect Prediction from Unrealistic Datacited by 5
  14. 2023A comprehensive review of automatic text summarization techniques: method, data, evaluation and codingcited by 4
  15. 2023Reviewer assignment problem: A scoping reviewcited by 4
  16. 2021A Survey of Knowledge Enhanced Pre-trained Modelscited by 3
  17. 2023A study on the impact of pre-trained model on Just-In-Time defect predictioncited by 3
  18. 2021Literature review on vulnerability detection using NLP technologycited by 2
  19. 2021A Survey On Neural Word Embeddingscited by 2
  20. 2024SetCSE: Set Operations using Contrastive Learning of Sentence Embeddingscited by 2
  21. 2021A Comprehensive Comparison of Word Embeddings in Event & Entity Coreference Resolutioncited by 1
  22. 2021Focusing on Potential Named Entities During Active Label Acquisitioncited by 1
  23. 2023Characterizing and Measuring Linguistic Dataset Driftcited by 1
  24. 2023A Comparative Analysis of Pretrained Language Models for Text-to-Speechcited by 1
  25. 2023Empowering ChatGPT-Like Large-Scale Language Models with Local Knowledge Base for Industrial Prognostics and Health Managementcited by 1
  26. 2024Engineering A Large Language Model From Scratchcited by 1
  27. 2025Med-gte-hybrid: A contextual embedding transformer model for extracting actionable information from clinical textscited by 1
  28. 2021Semantic maps and metrics for science Semantic maps and metrics for science using deep transformer encoderscited by 0
  29. 2021MigrationsKB: A Knowledge Base of Public Attitudes towards Migrations and their Driving Factorscited by 0
  30. 2021A Deep Learning Approach for Ontology Enrichment from Unstructured Textcited by 0
  31. 2022Word-Embeddings Distinguish Denominal and Root-Derived Verbs in Semiticcited by 0
  32. 2022A Graphical Model for Fusing Diverse Microbiome Datacited by 0
  33. 2025The cell as a token: high-dimensional geometry in language models and cell embeddingscited by 0
  34. 2025Contextual Embedding-based Clustering to Identify Topics for Healthcare Service Improvementcited by 0
  35. 2025Mechanistic Decomposition of Sentence Representationscited by 0
  36. 2025Sentiment Analysis in Learning Management Systems Understanding Student Feedback at Scalecited by 0