mlsanityData to 14 Oct 2026

An Efficient Self-Supervised Cross-View Training For Sentence Embedding

arXiv 2311.03228 ↗First posted 6 Nov 2023cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Self-supervised sentence representation learning is the task of constructing an embedding space for sentences without relying on human annotation efforts. One straightforward approach is to finetune a pretrained language model (PLM) with a representation learning method such as contrastive learning. While this approach achieves impressive performance on larger PLMs, the performance rapidly degrades as the number of parameters decreases. In this paper, we propose a framework called Self-supervised Cross-View Training (SCT) to narrow the performance gap between large and small PLMs. To evaluate the effectiveness of SCT, we compare it to 5 baseline and state-of-the-art competitors on seven Semantic Textual Similarity (STS) benchmarks using 5 PLMs with the number of parameters ranging from 4M to 340M. The experimental results show that STC outperforms the competitors for PLMs with less than 100M parameters in 18 of 21 cases.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 34 of its 49 references that are in the map. First posted on arXiv 6 Nov 2023.

Builds on

34 of 49 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  9. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  11. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  12. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
22 more
  1. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  2. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  3. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 430
  4. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  5. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  6. 2020CERT: Contrastive Self-supervised Learning for Language Understandingcited by 169
  7. 2021SEED: Self-supervised Distillation For Visual Representationcited by 134
  8. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 132
  9. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
  10. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  11. 2020An Unsupervised Sentence Embedding Method by Mutual Information Maximizationcited by 91
  12. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  13. 2021ReSSL: Relational Self-Supervised Learning with Weak Augmentationcited by 84
  14. 2020On the Sentence Embeddings from Pre-trained Language Modelscited by 83
  15. 2022How Does SimSiam Avoid Collapse Without Negative Samples? A Unified Understanding with Self-supervised Contrastive Learningcited by 65
  16. 2022Debiased Contrastive Learning of Unsupervised Sentence Representationscited by 56
  17. 2021Exploring the Equivalence of Siamese Self-Supervised Learning via A Unified Gradient Frameworkcited by 49
  18. 2020ISD: Self-Supervised Learning by Iterative Similarity Distillationcited by 33
  19. 2015Semi-supervised Question Retrieval with Gated Convolutionscited by 28
  20. 2022Exploring the Impact of Negative Samples of Contrastive Learning: A Case Study of Sentence Embeddingcited by 25
  21. 2021Virtual Augmentation Supported Contrastive Learning of Sentence Representationscited by 21
  22. 2022SCD: Self-Contrastive Decorrelation for Sentence Embeddingscited by 20

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Space Decomposition for Sentence Embeddingcited by 1