Non-contrastive sentence representations via self-supervision
Not on the map. Why?
Abstract (arXiv)
Sample contrastive methods, typically referred to simply as contrastive are the foundation of most unsupervised methods to learn text and sentence embeddings. On the other hand, a different class of self-supervised loss functions and methods have been considered in the computer vision community and referred to as dimension contrastive. In this paper, we thoroughly compare this class of methods with the standard baseline for contrastive sentence embeddings, SimCSE. We find that self-supervised embeddings trained using dimension contrastive objectives can outperform SimCSE on downstream tasks without needing auxiliary loss functions.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 31 of its 51 references that are in the map. First posted on arXiv 26 Oct 2023.
Builds on
31 of 51 references.
References · 31
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
- 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
- 2015A large annotated corpus for learning natural language inferencecited by 1,837
- 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
- 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
- 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
- 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
- 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
- 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
- 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
19 more
- 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
- 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
- 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 426
- 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
- 2021A Survey of Data Augmentation Approaches for NLPcited by 408
- 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
- 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
- 2020MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmarkcited by 156
- 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
- 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 131
- 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
- 2020Neural CRF Model for Sentence Alignment in Text Simplificationcited by 106
- 2017A Continuously Growing Dataset of Sentential Paraphrasescited by 72
- 2021I Wish I Would Have Loved This One, But I Didn't -- A Multilingual Dataset for Counterfactual Detection in Product Reviewscited by 44
- 2021TWEAC: Transformer with Extendable QA Agent Classifierscited by 37
- 2018Adversarial Domain Adaptation for Duplicate Question Detectioncited by 30
- 2022InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddingscited by 25
- 2022SCD: Self-Contrastive Decorrelation for Sentence Embeddingscited by 19
- 2023SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectivescited by 15
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.