mlsanityData to 7 Oct 2026

Sen2Pro: A Probabilistic Perspective to Sentence Embedding from Pre-trained Language Model

arXiv 2306.02247 ↗First posted 4 Jun 2023cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Sentence embedding is one of the most fundamental tasks in Natural Language Processing and plays an important role in various tasks. The recent breakthrough in sentence embedding is achieved by pre-trained language models (PLMs). Despite its success, an embedded vector (Sen2Vec) representing a point estimate does not naturally express uncertainty in a taskagnostic way. This paper thereby proposes an efficient framework on probabilistic sentence embedding (Sen2Pro) from PLMs, and it represents a sentence as a probability density distribution in an embedding space to reflect both model uncertainty and data uncertainty (i.e., many-to-one nature) in the sentence representation. The proposed framework performs in a plug-and-play way without retraining PLMs anymore, and it is easy to implement and generally applied on top of any PLM. The superiority of Sen2Pro over Sen2Vec has been theoretically verified and practically illustrated on different NLP tasks.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 39 of its 70 references that are in the map. First posted on arXiv 4 Jun 2023.

Builds on

39 of 70 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  7. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  8. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  9. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  10. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  11. 2018Deep contextualized word representationscited by 2,285
  12. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
27 more
  1. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  2. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  3. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
  4. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
  5. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
  6. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  7. 2015Skip-Thought Vectorscited by 321
  8. 2019On Mutual Information Maximization for Representation Learningcited by 315
  9. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  10. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 198
  11. 2019The Second Conversational Intelligence Challenge (ConvAI2)cited by 178
  12. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  13. 2017Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responsescited by 98
  14. 2014Word Representations via Gaussian Embeddingcited by 96
  15. 2020An Unsupervised Sentence Embedding Method by Mutual Information Maximizationcited by 91
  16. 2016Learning Distributed Representations of Sentences from Unlabelled Datacited by 90
  17. 2021WhiteningBERT: An Easy Unsupervised Sentence Embedding Approachcited by 68
  18. 2015Towards Universal Paraphrastic Sentence Embeddingscited by 59
  19. 2017Dynamic Word Embeddingscited by 51
  20. 2021CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understandingcited by 50
  21. 2018From Word to Sense Embeddings: A Survey on Vector Representations of Meaningcited by 49
  22. 2022On the Evaluation Metrics for Paraphrase Generationcited by 32
  23. 2022Just Rank: Rethinking Evaluation with Word and Sentence Similaritiescited by 20
  24. 2018Rate Distortion For Model Compression: From Theory To Practicecited by 14
  25. 2019Density Matching for Bilingual Word Embeddingcited by 11
  26. 2020Phase Transitions in Rate Distortion Theory and Deep Learningcited by 5
  27. 2014Convex Banding of the Covariance Matrixcited by 4

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Uncertainty-driven Embedding Convolutioncited by 0