mlsanityData to 14 Oct 2026

A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the ChatGPT Era and Beyond

arXiv 2305.12641 ↗First posted 22 May 2023cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Sentence representations are a critical component in NLP applications such as retrieval, question answering, and text classification. They capture the meaning of a sentence, enabling machines to understand and reason over human language. In recent years, significant progress has been made in developing methods for learning sentence representations, including unsupervised, supervised, and transfer learning approaches. However there is no literature review on sentence representations till now. In this paper, we provide an overview of the different methods for sentence representation learning, focusing mostly on deep learning models. We provide a systematic organization of the literature, highlighting the key contributions and challenges in this area. Overall, our review highlights the importance of this area in natural language processing, the progress made in sentence representation learning, and the challenges that remain. We conclude with directions for future research, suggesting potential avenues for improving the quality and efficiency of sentence representations.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 66 of its 82 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

66 of 82 references.

References · 66

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  9. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  11. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
38 more
  1. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  2. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  3. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  4. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  5. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  6. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 463
  7. 2021Large Dual Encoders Are Generalizable Retrieverscited by 456
  8. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 432
  9. 2015Skip-Thought Vectorscited by 322
  10. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  11. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 251
  12. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  13. 2021Generating Datasets with Pretrained Language Modelscited by 193
  14. 2020Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Taskscited by 138
  15. 2022Efficient Few-Shot Learning Without Promptscited by 137
  16. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 132
  17. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
  18. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  19. 2020An Unsupervised Sentence Embedding Method by Mutual Information Maximizationcited by 91
  20. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  21. 2021Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Modelscited by 78
  22. 2021ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embeddingcited by 74
  23. 2021WhiteningBERT: An Easy Unsupervised Sentence Embedding Approachcited by 69
  24. 2022Debiased Contrastive Learning of Unsupervised Sentence Representationscited by 56
  25. 2022Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusterscited by 44
  26. 2020Universal Sentence Representation Learning with Conditional Masked Language Modelcited by 34
  27. 2022Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learningcited by 30
  28. 2022PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddingscited by 29
  29. 2022InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddingscited by 26
  30. 2022Exploring the Impact of Negative Samples of Contrastive Learning: A Case Study of Sentence Embeddingcited by 25
  31. 2021Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained Language Modelscited by 22
  32. 2022MCSE: Multimodal Contrastive Learning of Sentence Embeddingscited by 22
  33. 2021Virtual Augmentation Supported Contrastive Learning of Sentence Representationscited by 21
  34. 2022EASE: Entity-Aware Contrastive Learning of Sentence Embeddingcited by 21
  35. 2022SCD: Self-Contrastive Decorrelation for Sentence Embeddingscited by 20
  36. 2022Generate, Discriminate and Contrast: A Semi-Supervised Sentence Representation Learning Frameworkcited by 19
  37. 2022Learning Dialogue Representations from Consecutive Utterancescited by 17
  38. 2019A Bilingual Generative Transformer for Semantic Sentence Embeddingcited by 16

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 4; 2026: 4.

12 papers.

  1. 2024Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmarkcited by 19
  2. 2024An Empirical Study on Capability of Large Language Models in Understanding Code Semanticscited by 7
  3. 2026Reasoning-Driven Synthetic Data Generation and Evaluationcited by 5
  4. 2025ScaleDoc: Scaling LLM-based Predicates over Large Document Collectionscited by 3
  5. 2025A Retrieval-Based Approach to Medical Procedure Matching in Romaniancited by 2
  6. 2025Dense Retrievers Can Fail on Simple Queries: Revealing The Granularity Dilemma of Embeddingscited by 2
  7. 2023Translation Aligned Sentence Embeddings for Turkish Languagecited by 0
  8. 2024DefSent+: Improving sentence embeddings of language models by projecting definition sentences into a quasi-isotropic or isotropic vector space of unlimited dictionary entriescited by 0
  9. 2025Collaborative Intelligence: Topic Modelling of Large Language Model use in Live Cybersecurity Operationscited by 0
  10. 2026TALAS: Teacher-Anchored Layer Alignment with Adaptive Sharpness-Aware Minimization for Embedding Distillationcited by 0
  11. 2026From transcription to semantic corpus analysis: unsupervised learning of sentence representations for ancient languagescited by 0
  12. 2026Know Yourself and Know Your Neighbour: A Syntactically Informed Self-Supervised Compositional Sentence Representation Learning Framework using a Recursive Hypernetworkcited by 0