mlsanityData to 7 Oct 2026

Contrastive Learning with Prompt-derived Virtual Semantic Prototypes for Unsupervised Sentence Embedding

arXiv 2211.03348 ↗First posted 7 Nov 2022cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Contrastive learning has become a new paradigm for unsupervised sentence embeddings. Previous studies focus on instance-wise contrastive learning, attempting to construct positive pairs with textual data augmentation. In this paper, we propose a novel Contrastive learning method with Prompt-derived Virtual semantic Prototypes (ConPVP). Specifically, with the help of prompts, we construct virtual semantic prototypes to each instance, and derive negative prototypes by using the negative form of the prompts. Using a prototypical contrastive loss, we enforce the anchor sentence embedding to be close to its corresponding semantic prototypes, and far apart from the negative prototypes as well as the prototypes of other sentences. Extensive experimental results on semantic textual similarity, transfer, and clustering tasks demonstrate the effectiveness of our proposed model compared to strong baselines. Code is available at https://github.com/lemon0830/promptCSE.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 42 of its 66 references that are in the map. First posted on arXiv 7 Nov 2022.

Builds on

42 of 66 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  11. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  12. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
30 more
  1. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  2. 2021GPT Understands, Toocited by 715
  3. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  4. 2015Skip-Thought Vectorscited by 321
  5. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  6. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  7. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  8. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 198
  9. 2018An efficient framework for learning sentence representationscited by 197
  10. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  11. 2021Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classificationcited by 183
  12. 2021Template-Based Named Entity Recognition Using BARTcited by 153
  13. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  14. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
  15. 2017Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Featurescited by 103
  16. 2020An Unsupervised Sentence Embedding Method by Mutual Information Maximizationcited by 91
  17. 2016Learning Distributed Representations of Sentences from Unlabelled Datacited by 90
  18. 2020Few-Shot Text Generation with Pattern-Exploiting Trainingcited by 87
  19. 2022PromptBERT: Improving BERT Sentence Embeddings with Promptscited by 75
  20. 2021ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embeddingcited by 73
  21. 2021CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understandingcited by 50
  22. 2022SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with Soft Negative Samplescited by 39
  23. 2017Self-Taught Convolutional Neural Networks for Short Text Clusteringcited by 34
  24. 2021Prototypical Representation Learning for Relation Extractioncited by 33
  25. 2021Pairwise Supervised Contrastive Learning of Sentence Representationscited by 31
  26. 2015Text Understanding from Scratchcited by 25
  27. 2020Clustering based Contrastive Learning for Improving Face Representationscited by 21
  28. 2021Generating Disentangled Arguments with Prompts: A Simple Event Extraction Framework that Workscited by 6
  29. 2022Eliciting Knowledge from Pretrained Language Models for Prototypical Prompt Verbalizercited by 6
  30. 2021CLLD: Contrastive Learning with Label Distance for Text Classificationcited by 1

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 1.

6 papers.

  1. 2023Contrastive Learning of Sentence Embeddings from Scratchcited by 24
  2. 2023A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the ChatGPT Era and Beyondcited by 11
  3. 2023CoT-BERT: Enhancing Unsupervised Sentence Representation through Chain-of-Thoughtcited by 8
  4. 2023Clustering-Aware Negative Sampling for Unsupervised Sentence Representationcited by 5
  5. 2024Prototypical Extreme Multi-label Classification with a Dynamic Margin Losscited by 2
  6. 2023Adapting Sentence Transformers for the Aviation Domaincited by 1