mlsanityData to 7 Oct 2026

Self-Guided Contrastive Learning for BERT Sentence Representations

arXiv 2106.07345 ↗First posted 3 Jun 2021cs.CL cs.AIcited by 103

Not on the map. Why?

Abstract (arXiv)

Although BERT and its variants have reshaped the NLP landscape, it still remains unclear how best to derive sentence embeddings from such pre-trained Transformers. In this work, we propose a contrastive learning method that utilizes self-guidance for improving the quality of BERT sentence representations. Our method fine-tunes BERT in a self-supervised fashion, does not rely on data augmentation, and enables the usual [CLS] token embeddings to function as sentence vectors. Moreover, we redesign the contrastive learning objective (NT-Xent) and apply it to sentence representation learning. We demonstrate with extensive experiments that our approach is more effective than competitive baselines on diverse sentence-related tasks. We also show it is efficient at inference and robust to domain shifts.

Key facts. Cited by 103 papers in the map (data through 7 Oct 2026). Builds on 35 of its 48 references that are in the map. First posted on arXiv 3 Jun 2021.

Builds on

35 of 48 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  9. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  12. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
23 more
  1. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  2. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  3. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  4. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 918
  5. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  6. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  7. 2020A Survey on Contrastive Self-supervised Learningcited by 553
  8. 2018Universal Sentence Encodercited by 485
  9. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  10. 2020Keystroke Dynamics as Part of Lifeloggingcited by 334
  11. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  12. 2020Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningcited by 275
  13. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
  14. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
  15. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  16. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  17. 2018An efficient framework for learning sentence representationscited by 197
  18. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  19. 2019Facebook FAIR's WMT19 News Translation Task Submissioncited by 179
  20. 2020CERT: Contrastive Self-supervised Learning for Language Understandingcited by 169
  21. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 156
  22. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
  23. 2020SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Modelscited by 55

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 34; 2023: 35; 2024: 16; 2025: 7; 2026: 3.

103 papers.

Sort:

  1. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 131
  2. 2022Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddingscited by 75
  3. 2021ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embeddingcited by 73
  4. 2022Debiased Contrastive Learning of Unsupervised Sentence Representationscited by 56
  5. 2022Incorporating Hierarchy into Text Encoder: a Contrastive Learning Approach for Hierarchical Text Classificationcited by 52
  6. 2024The Double-Ellipsoid Geometry of CLIPcited by 42
  7. 2021Contrast and Generation Make BART a Good Dialogue Emotion Recognizercited by 35
  8. 2021TaCL: Improving BERT Pre-training with Token-aware Contrastive Learningcited by 32
  9. 2022Learning from the Dictionary: Heterogeneous Knowledge Guided Fine-tuning for Chinese Spell Checkingcited by 31
  10. 2022Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learningcited by 30
  11. 2022PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddingscited by 29
  12. 2022New Intent Discovery with Pre-training and Contrastive Learningcited by 28
38 more
  1. 2022Contrastive Learning with Hard Negative Entities for Entity Set Expansioncited by 26
  2. 2022English Contrastive Learning Can Learn Universal Cross-lingual Sentence Embeddingscited by 26
  3. 2022InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddingscited by 25
  4. 2022Graph Enhanced Contrastive Learning for Radiology Findings Summarizationcited by 24
  5. 2023RankCSE: Unsupervised Sentence Representations Learning via Learning to Rankcited by 23
  6. 2022MCSE: Multimodal Contrastive Learning of Sentence Embeddingscited by 22
  7. 2023Facilitating Contrastive Learning of Discourse Relational Senses by Exploiting the Hierarchy of Sense Relationscited by 22
  8. 2021Virtual Augmentation Supported Contrastive Learning of Sentence Representationscited by 21
  9. 2022EASE: Entity-Aware Contrastive Learning of Sentence Embeddingcited by 21
  10. 2023Part Aware Contrastive Learning for Self-Supervised Action Recognitioncited by 21
  11. 2023Whitening-based Contrastive Learning of Sentence Embeddingscited by 21
  12. 2021Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillationscited by 20
  13. 2023Weakly Supervised Detection of Hallucinations in LLM Activationscited by 20
  14. 2022Making Pre-trained Language Models End-to-end Few-shot Learners with Contrastive Prompt Tuningcited by 19
  15. 2021MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Modelscited by 18
  16. 2022ContraCLM: Contrastive Learning For Causal Language Modelcited by 18
  17. 2023Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similaritycited by 17
  18. 2023Alleviating Over-smoothing for Unsupervised Sentence Representationcited by 16
  19. 2022Ranking-Enhanced Unsupervised Sentence Representation Learningcited by 15
  20. 2023SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectivescited by 15
  21. 2023Multi-level Contrastive Learning for Script-based Character Understandingcited by 15
  22. 2023Cluster-Level Contrastive Learning for Emotion Recognition in Conversationscited by 14
  23. 2022Fine-grained Category Discovery under Coarse-grained supervision with Hierarchical Weighted Self-contrastive Learningcited by 13
  24. 2024KDMCSE: Knowledge Distillation Multimodal Sentence Embeddings with Adaptive Angular margin Contrastive Learningcited by 13
  25. 2022Multilingual Representation Distillation with Contrastive Learningcited by 11
  26. 2023Evaluating Embedding APIs for Information Retrievalcited by 11
  27. 2023A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the ChatGPT Era and Beyondcited by 11
  28. 2023Language Models are Universal Embedderscited by 11
  29. 2022Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddingscited by 10
  30. 2022Enhancing Out-of-Distribution Detection in Natural Language Understanding via Implicit Layer Ensemblecited by 9
  31. 2022SDA: Simple Discrete Augmentation for Contrastive Sentence Representation Learningcited by 8
  32. 2024Hyper-CL: Conditioning Sentence Representations with Hypernetworkscited by 8
  33. 2024FairDgcl: Fairness-aware Recommendation with Dynamic Graph Contrastive Learningcited by 8
  34. 2022Contrastive Demonstration Tuning for Pre-trained Language Modelscited by 7
  35. 2022miCSE: Mutual Information Contrastive Learning for Low-shot Sentence Embeddingscited by 7
  36. 2023Joint Multiple Intent Detection and Slot Filling with Supervised Contrastive Learning and Self-Distillationcited by 7
  37. 2023A Diffusion Weighted Graph Framework for New Intent Discoverycited by 7
  38. 2024Multilingual Sentence-T5: Scalable Sentence Encoders for Multilingual Applicationscited by 7

50 most cited of 103 All 103 (JSON)