mlsanityData to 7 Oct 2026

A Survey of Text Representation Methods and Their Genealogy

arXiv 2211.14591 ↗First posted 26 Nov 2022cs.CL cs.AI cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

In recent years, with the advent of highly scalable artificial-neural-network-based text representation methods the field of natural language processing has seen unprecedented growth and sophistication. It has become possible to distill complex linguistic information of text into multidimensional dense numeric vectors with the use of the distributional hypothesis. As a consequence, text representation methods have been evolving at such a quick pace that the research community is struggling to retain knowledge of the methods and their interrelations. We contribute threefold to this lack of compilation, composition, and systematization by providing a survey of current approaches, by arranging them in a genealogy, and by conceptualizing a taxonomy of text representation methods to examine and explain the state-of-the-art. Our research is a valuable guide and reference for artificial intelligence researchers and practitioners interested in natural language processing applications such as recommender systems, chatbots, and sentiment analysis.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 48 of its 107 references that are in the map. First posted on arXiv 26 Nov 2022.

Builds on

48 of 107 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  10. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  11. 2020Longformer: The Long-Document Transformercited by 3,362
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
36 more
  1. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  3. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  4. 2018Deep contextualized word representationscited by 2,285
  5. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  9. 2020Reformer: The Efficient Transformercited by 1,658
  10. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  12. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  13. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  14. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  15. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  16. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  17. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  18. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  19. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  20. 2016FastText.zip: Compressing text classification modelscited by 331
  21. 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
  22. 2015Skip-Thought Vectorscited by 321
  23. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  24. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 301
  25. 2017Recent Trends in Deep Learning Based Natural Language Processingcited by 262
  26. 2019ERNIE 2.0: A Continual Pre-training Framework for Language Understandingcited by 230
  27. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  28. 2020ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingcited by 221
  29. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
  30. 2017Learned in Translation: Contextualized Word Vectorscited by 104
  31. 2019Attention in Natural Language Processingcited by 100
  32. 2020A Survey on Text Classification: From Shallow to Deep Learningcited by 80
  33. 2018From Word to Sense Embeddings: A Survey on Vector Representations of Meaningcited by 49
  34. 2022A Survey of Knowledge-Intensive NLP with Pre-Trained Language Modelscited by 30
  35. 2016A Joint Model for Word Embedding and Word Morphologycited by 14
  36. 2017Symbolic, Distributed and Distributional Representations for Natural Language Processing in the Era of Deep Learning: a Surveycited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 3; 2026: 1.

5 papers.

  1. 2023Generative AIcited by 119
  2. 2025CareerBERT: Matching Resumes to ESCO Jobs in a Shared Embedding Space for Generic Job Recommendationscited by 5
  3. 2025Hate Speech and Sentiment of YouTube Video Comments From Public and Private Sources Covering the Israel-Palestine Conflictcited by 1
  4. 2025Innovative tokenisation of structured data for LLM trainingcited by 0
  5. 2026Mapping the Web of Science, a large-scale graph and text-based dataset with LLM embeddingscited by 0