# A Comprehensive Survey on Word Representation Models: From Classical to State-Of-The-Art Word Representation Language Models

arXiv: https://arxiv.org/abs/2010.15036 · HTML: https://mlsanity.com/paper/2010.15036/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Oct 2020.

## Built on by (most cited first, 30 of 30)

- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methods](https://mlsanity.com/paper/2403.02901/) (2024)
- [Large Language Models for Education: A Survey](https://mlsanity.com/paper/2405.13001/) (2024)
- [Investigating the use of ChatGPT for the scheduling of construction projects](https://mlsanity.com/paper/2302.02805/) (2023)
- [Algorithm Research of ELMo Word Embedding and Deep Learning Multimodal Transformer in Image Description](https://mlsanity.com/paper/2408.06357/) (2024)
- [Beware the Rationalization Trap! When Language Model Explainability Diverges from our Mental Models of Language](https://mlsanity.com/paper/2207.06897/) (2022)
- [Recommender systems based on graph embedding techniques: A comprehensive review](https://mlsanity.com/paper/2109.09587/) (2021)
- [Benchmarking for Public Health Surveillance tasks on Social Media with a Domain-Specific Pretrained Language Model](https://mlsanity.com/paper/2204.04521/) (2022)
- [Linguistically inspired roadmap for building biologically reliable protein language models](https://mlsanity.com/paper/2207.00982/) (2022)
- [MTEB-French: Resources for French Sentence Embedding Evaluation and Analysis](https://mlsanity.com/paper/2405.20468/) (2024)
- [Guided Semi-Supervised Non-negative Matrix Factorization on Legal Documents](https://mlsanity.com/paper/2201.13324/) (2022)
- [BikeBench: A Bicycle Design Benchmark for Generative Models with Objectives and Constraints](https://mlsanity.com/paper/2508.00830/) (2025)
- [Selecting Parallel In-domain Sentences for Neural Machine Translation Using Monolingual Texts](https://mlsanity.com/paper/2112.06096/) (2021)
- [Artificial Text Detection with Multiple Training Strategies](https://mlsanity.com/paper/2212.05194/) (2022)
- [Classifying vaccine sentiment tweets by modelling domain-specific representation and commonsense knowledge into context-aware attentive GRU](https://mlsanity.com/paper/2106.09589/) (2021)
- [Methods for Acquiring and Incorporating Knowledge into Stock Price Prediction: A Survey](https://mlsanity.com/paper/2308.04947/) (2023)
- [From Chatter to Matter: Addressing Critical Steps of Emotion Recognition Learning in Task-oriented Dialogue](https://mlsanity.com/paper/2308.12648/) (2023)
- [NLU-STR at SemEval-2024 Task 1: Generative-based Augmentation and Encoder-based Scoring for Semantic Textual Relatedness](https://mlsanity.com/paper/2405.00659/) (2024)
- [Effects of Parametric and Non-Parametric Methods on High Dimensional Sparse Matrix Representations](https://mlsanity.com/paper/2202.02894/) (2022)
- [Dual Embodied-Symbolic Concept Representations for Deep Learning](https://mlsanity.com/paper/2203.00600/) (2022)
- [TwERC: High Performance Ensembled Candidate Generation for Ads Recommendation at Twitter](https://mlsanity.com/paper/2302.13915/) (2023)
- [Automated Text Mining of Experimental Methodologies from Biomedical Literature](https://mlsanity.com/paper/2404.13779/) (2024)
- [Enhancing Suicide Risk Detection on Social Media through Semi-Supervised Deep Label Smoothing](https://mlsanity.com/paper/2405.05795/) (2024)
- [Performance Comparison of Simple Transformer and Res-CNN-BiLSTM for Cyberbullying Classification](https://mlsanity.com/paper/2206.02206/) (2022)
- [Persian Typographical Error Type Detection Using Deep Neural Networks on Algorithmically-Generated Misspellings](https://mlsanity.com/paper/2305.11731/) (2023)
- [Embedding Mental Health Discourse for Community Recommendation](https://mlsanity.com/paper/2307.03892/) (2023)
- [When Do Discourse Markers Affect Computational Sentence Understanding?](https://mlsanity.com/paper/2309.00368/) (2023)
- [An Embedded Diachronic Sense Change Model with a Case Study from Ancient Greek](https://mlsanity.com/paper/2311.00541/) (2023)
- [Natural Language Processing for Electronic Health Records in Scandinavian Languages: Norwegian, Swedish, and Danish](https://mlsanity.com/paper/2503.18539/) (2025)
- [Innovative tokenisation of structured data for LLM training](https://mlsanity.com/paper/2508.01685/) (2025)

## Abstract (arXiv)

Word representation has always been an important research area in the history of natural language processing (NLP). Understanding such complex text data is imperative, given that it is rich in information and can be used widely across various applications. In this survey, we explore different word representation models and its power of expression, from the classical to modern-day state-of-the-art word representation language models (LMS). We describe a variety of text representation methods, and model designs have blossomed in the context of NLP, including SOTA LMs. These models can transform large volumes of text into effective vector representations capturing the same semantic information. Further, such representations can be utilized by various machine learning (ML) algorithms for a variety of NLP related tasks. In the end, this survey briefly discusses the commonly used ML and DL based classifiers, evaluation metrics and the applications of these word embeddings in different NLP tasks.