mlsanityData to 14 Oct 2026

ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer

arXiv 2105.11741 ↗First posted 25 May 2021cs.CL cs.AIcited by 231

Not on the map. Why?

Abstract (arXiv)

Learning high-quality sentence representations benefits a wide range of natural language processing tasks. Though BERT-based pre-trained language models achieve high performance on many downstream tasks, the native derived sentence representations are proved to be collapsed and thus produce a poor performance on the semantic textual similarity (STS) tasks. In this paper, we present ConSERT, a Contrastive Framework for Self-Supervised Sentence Representation Transfer, that adopts contrastive learning to fine-tune BERT in an unsupervised and effective way. By making use of unlabeled texts, ConSERT solves the collapse issue of BERT-derived sentence representations and make them more applicable for downstream tasks. Experiments on STS datasets demonstrate that ConSERT achieves an 8% relative improvement over the previous state-of-the-art, even comparable to the supervised SBERT-NLI. And when further incorporating NLI supervision, we achieve new state-of-the-art performance on STS tasks. Moreover, ConSERT obtains comparable results with only 1000 samples available, showing its robustness in data scarcity scenarios.

Key facts. Cited by 231 papers in the map (data through 14 Oct 2026). Builds on 27 of its 39 references that are in the map. First posted on arXiv 25 May 2021.

Builds on

27 of 39 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  9. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  10. 2016Adversarial examples in the physical worldcited by 1,769
  11. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  12. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
15 more
  1. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  2. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 432
  3. 2015Skip-Thought Vectorscited by 322
  4. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  5. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 255
  6. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 251
  7. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  8. 2020CERT: Contrastive Self-supervised Learning for Language Understandingcited by 169
  9. 2016Learning Distributed Representations of Sentences from Unlabelled Datacited by 91
  10. 2020An Unsupervised Sentence Embedding Method by Mutual Information Maximizationcited by 91
  11. 2020A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generationcited by 72
  12. 2016Adversarial Diversity and Hard Positive Generationcited by 35
  13. 2020Universal Sentence Representation Learning with Conditional Masked Language Modelcited by 34
  14. 2020SLM: Learning a Discourse Language Representation with Sentence Unshufflingcited by 24
  15. 2020Cross-Thought for Sentence Encoder Pre-trainingcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 17; 2022: 81; 2023: 71; 2024: 42; 2025: 14; 2026: 6.

231 papers.

Sort:

By area:

  1. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 132
  2. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  3. 2021A Survey on Data Augmentation for Text Classificationcited by 111
  4. 2023A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directionscited by 105
  5. 2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99
  6. 2023AnglE-optimized Text Embeddingscited by 99
  7. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  8. 2022PromptBERT: Improving BERT Sentence Embeddings with Promptscited by 75
  9. 2021ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embeddingcited by 74
  10. 2021SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representationcited by 63
  11. 2024Anisotropy Is Inherent to Self-Attention in Transformerscited by 63
  12. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
38 more
  1. 2022Self-supervised Trajectory Representation Learning with Temporal Regularities and Travel Semanticscited by 58
  2. 2022Debiased Contrastive Learning of Unsupervised Sentence Representationscited by 56
  3. 2022Detect Rumors in Microblog Posts for Low-Resource Domains via Adversarial Contrastive Learningcited by 41
  4. 2024GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuningcited by 40
  5. 2022SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with Soft Negative Samplescited by 39
  6. 2021Contrast and Generation Make BART a Good Dialogue Emotion Recognizercited by 35
  7. 2022SPACE-3: Unified Dialog Model Pre-training for Task-Oriented Dialog Understanding and Generationcited by 34
  8. 2023Text-CRS: A Generalized Certified Robustness Framework against Textual Adversarial Attackscited by 34
  9. 2021TaCL: Improving BERT Pre-training with Token-aware Contrastive Learningcited by 32
  10. 2023BeLLM: Backward Dependency Enhanced Large Language Model for Sentence Embeddingscited by 32
  11. 2022Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learningcited by 30
  12. 2022PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddingscited by 29
  13. 2023Improving Contrastive Learning of Sentence Embeddings from AI Feedbackcited by 29
  14. 2022New Intent Discovery with Pre-training and Contrastive Learningcited by 28
  15. 2022Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embeddingcited by 28
  16. 2022SPACE-2: Tree-Structured Semi-Supervised Contrastive Pre-training for Task-Oriented Dialog Understandingcited by 28
  17. 2023Scene Graph as Pivoting: Inference-time Image-free Unsupervised Multimodal Machine Translation with Visual Scene Hallucinationcited by 27
  18. 2023Pushing the Limits of ChatGPT on NLP Taskscited by 27
  19. 2022InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddingscited by 26
  20. 2022English Contrastive Learning Can Learn Universal Cross-lingual Sentence Embeddingscited by 26
  21. 2022Exploring the Impact of Negative Samples of Contrastive Learning: A Case Study of Sentence Embeddingcited by 25
  22. 2023Cross-modal Contrastive Learning for Multimodal Fake News Detectioncited by 23
  23. 2023RankCSE: Unsupervised Sentence Representations Learning via Learning to Rankcited by 23
  24. 2022MCSE: Multimodal Contrastive Learning of Sentence Embeddingscited by 22
  25. 2023Facilitating Contrastive Learning of Discourse Relational Senses by Exploiting the Hierarchy of Sense Relationscited by 22
  26. 2021Virtual Augmentation Supported Contrastive Learning of Sentence Representationscited by 21
  27. 2022CodeRetriever: Unimodal and Bimodal Contrastive Learning for Code Searchcited by 21
  28. 2022CLMLF:A Contrastive Learning and Multi-Layer Fusion Method for Multimodal Sentiment Detectioncited by 21
  29. 2022EASE: Entity-Aware Contrastive Learning of Sentence Embeddingcited by 21
  30. 2022COCOA: Cross Modality Contrastive Learning for Sensor Datacited by 21
  31. 2023Whitening-based Contrastive Learning of Sentence Embeddingscited by 21
  32. 2024LLMvsSmall Model? Large Language Model Based Text Augmentation Enhanced Personality Detection Modelcited by 21
  33. 2021Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillationscited by 20
  34. 2022Just Rank: Rethinking Evaluation with Word and Sentence Similaritiescited by 20
  35. 2021DialogueCSE: Dialogue-based Contrastive Learning of Sentence Embeddingscited by 19
  36. 2022Fine-tuning Pre-trained Language Models for Few-shot Intent Detection: Supervised Pre-training and Isotropizationcited by 19
  37. 2022E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generationcited by 19
  38. 2021MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Modelscited by 18

50 most cited of 231 All 231 (JSON)