mlsanityData to 7 Oct 2026

PromptBERT: Improving BERT Sentence Embeddings with Prompts

arXiv 2201.04337 ↗First posted 12 Jan 2022cs.CLcited by 75

Not on the map. Why?

Abstract (arXiv)

We propose PromptBERT, a novel contrastive learning method for learning better sentence representation. We firstly analyze the drawback of current sentence embedding from original BERT and find that it is mainly due to the static token embedding bias and ineffective BERT layers. Then we propose the first prompt-based sentence embeddings method and discuss two prompt representing methods and three prompt searching methods to make BERT achieve better sentence embeddings. Moreover, we propose a novel unsupervised training objective by the technology of template denoising, which substantially shortens the performance gap between the supervised and unsupervised settings. Extensive experiments show the effectiveness of our method. Compared to SimCSE, PromptBert achieves 2.29 and 2.58 points of improvement based on BERT and RoBERTa in the unsupervised setting.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Builds on 19 of its 34 references that are in the map. First posted on arXiv 12 Jan 2022.

Builds on

19 of 34 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  7. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  8. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  9. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  10. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  11. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  12. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
7 more
  1. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  2. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 250
  3. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  4. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 198
  5. 2020An Unsupervised Sentence Embedding Method by Mutual Information Maximizationcited by 91
  6. 2020On the Sentence Embeddings from Pre-trained Language Modelscited by 83
  7. 2021DefSent: Sentence Embeddings using Definition Sentencescited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 14; 2023: 26; 2024: 20; 2025: 10; 2026: 5.

75 papers.

Sort:

By area:

  1. 2024E5-V: Universal Embeddings with Multimodal Large Language Modelscited by 123
  2. 2023AnglE-optimized Text Embeddingscited by 98
  3. 2023Scaling Sentence Embeddings with Large Language Modelscited by 83
  4. 2023Language models are not naysayers: An analysis of language models on negation benchmarkscited by 73
  5. 2024Can GNN be Good Adapter for LLMs?cited by 71
  6. 2024Repetition Improves Language Model Embeddingscited by 70
  7. 2022Problems with Cosine as a Measure of Embedding Similarity for High Frequency Wordscited by 49
  8. 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Freecited by 37
  9. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  10. 2022PANDA: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptationcited by 31
  11. 2022Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learningcited by 30
  12. 2023Improving Contrastive Learning of Sentence Embeddings from AI Feedbackcited by 29
38 more
  1. 2024Simple Techniques for Enhancing Sentence Embeddings in Generative Language Modelscited by 26
  2. 2023Contrastive Learning of Sentence Embeddings from Scratchcited by 24
  3. 2022E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generationcited by 19
  4. 2022Trial2Vec: Zero-Shot Clinical Trial Document Similarity Search using Self-Supervisioncited by 18
  5. 2022Generate, Discriminate and Contrast: A Semi-Supervised Sentence Representation Learning Frameworkcited by 18
  6. 2023Meaning Representations from Trajectories in Autoregressive Modelscited by 18
  7. 2022Improving Contrastive Learning of Sentence Embeddings with Case-Augmented Positives and Retrieved Negativescited by 16
  8. 2024Prompt-based Code Completion via Multi-Retrieval Augmented Generationcited by 16
  9. 2022Ranking-Enhanced Unsupervised Sentence Representation Learningcited by 15
  10. 2022Prompt-based Connective Prediction Method for Fine-grained Implicit Discourse Relation Recognitioncited by 14
  11. 2024Take Care of Your Prompt Bias! Investigating and Mitigating Prompt Bias in Factual Knowledge Extractioncited by 13
  12. 2024Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Featurescited by 11
  13. 2022Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddingscited by 10
  14. 2022Differentiable Data Augmentation for Contrastive Sentence Representation Learningcited by 9
  15. 2023CoT-BERT: Enhancing Unsupervised Sentence Representation through Chain-of-Thoughtcited by 8
  16. 2024Extracting Sentence Embeddings from Pretrained Transformer Modelscited by 7
  17. 2022Contrastive Learning with Prompt-derived Virtual Semantic Prototypes for Unsupervised Sentence Embeddingcited by 6
  18. 2023Ditto: A Simple and Efficient Approach to Improve Sentence Embeddingscited by 6
  19. 2023From Alignment to Entailment: A Unified Textual Entailment Framework for Entity Alignmentcited by 6
  20. 2023Clustering-Aware Negative Sampling for Unsupervised Sentence Representationcited by 5
  21. 2023Revealing the Blind Spot of Sentence Encoder Evaluation by HEROScited by 5
  22. 2023DebCSE: Rethinking Unsupervised Contrastive Sentence Embedding Learning in the Debiasing Perspectivecited by 5
  23. 2024DenoSent: A Denoising Objective for Self-Supervised Sentence Representation Learningcited by 4
  24. 2024Interpretable Measures of Conceptual Similarity by Complexity-Constrained Descriptive Auto-Encodingcited by 4
  25. 2024Enhancing Unsupervised Sentence Embeddings via Knowledge-Driven Data Augmentation and Gaussian-Decayed Contrastive Learningcited by 4
  26. 2024Instruction Embedding: Latent Representations of Instructions Towards Task Identificationcited by 4
  27. 2023VendorLink: An NLP approach for Identifying & Linking Vendor Migrants & Potential Aliases on Darknet Marketscited by 3
  28. 2023Instance Smoothed Contrastive Learning for Unsupervised Sentence Embeddingcited by 3
  29. 2025CSE-SFP: Enabling Unsupervised Sentence Representation Learning via a Single Forward Passcited by 3
  30. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  31. 2022Block-SCL: Blocking Matters for Supervised Contrastive Learning in Product Matchingcited by 2
  32. 2023Weighted Sampling for Masked Language Modelingcited by 2
  33. 2023C-STS: Conditional Semantic Textual Similaritycited by 2
  34. 2024SetCSE: Set Operations using Contrastive Learning of Sentence Embeddingscited by 2
  35. 2025DEUCE: Dual-diversity Enhancement and Uncertainty-awareness for Cold-start Active Learningcited by 2
  36. 2025PromptAL: Sample-Aware Dynamic Soft Prompts for Few-Shot Active Learningcited by 2
  37. 2025CoDiEmb: A Collaborative yet Distinct Framework for Unified Representation Learning in Information Retrieval and Semantic Textual Similaritycited by 2
  38. 2026An Efficient and Effective Evaluator for Text2SQL Models on Unseen and Unlabeled Datacited by 2

50 most cited of 75 All 75 (JSON)