mlsanityData to 7 Oct 2026

UNSEE: Unsupervised Non-contrastive Sentence Embeddings

arXiv 2401.15316 ↗First posted 27 Jan 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We present UNSEE: Unsupervised Non-Contrastive Sentence Embeddings, a novel approach that outperforms SimCSE in the Massive Text Embedding benchmark. Our exploration begins by addressing the challenge of representation collapse, a phenomenon observed when contrastive objectives in SimCSE are replaced with non-contrastive objectives. To counter this issue, we propose a straightforward solution known as the target network, effectively mitigating representation collapse. The introduction of the target network allows us to leverage non-contrastive objectives, maintaining training stability while achieving performance improvements comparable to contrastive objectives. Our method has achieved peak performance in non-contrastive sentence embeddings through meticulous fine-tuning and optimization. This comprehensive effort has yielded superior sentence representation models, showcasing the effectiveness of our approach.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 36 of its 44 references that are in the map. First posted on arXiv 27 Jan 2024.

Builds on

36 of 44 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  11. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,837
24 more
  1. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  2. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  3. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  4. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  5. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  6. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  7. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  8. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  9. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  10. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  11. 2015Skip-Thought Vectorscited by 321
  12. 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
  13. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  14. 2020Whitening for Self-Supervised Representation Learningcited by 278
  15. 2018An efficient framework for learning sentence representationscited by 197
  16. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 131
  17. 2016Learning Distributed Representations of Sentences from Unlabelled Datacited by 90
  18. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  19. 2022PromptBERT: Improving BERT Sentence Embeddings with Promptscited by 75
  20. 2022Self-Supervised Learning with an Information Maximization Criterioncited by 38
  21. 2022Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learningcited by 30
  22. 2021Self-Supervised Classification Networkcited by 27
  23. 2021Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillationscited by 20
  24. 2022SCD: Self-Contrastive Decorrelation for Sentence Embeddingscited by 19

Built on by

Not cited yet.