mlsanityData to 7 Oct 2026

Alleviating Over-smoothing for Unsupervised Sentence Representation

arXiv 2305.06154 ↗First posted 9 May 2023cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

Currently, learning better unsupervised sentence representations is the pursuit of many natural language processing communities. Lots of approaches based on pre-trained language models (PLMs) and contrastive learning have achieved promising results on this task. Experimentally, we observe that the over-smoothing problem reduces the capacity of these powerful PLMs, leading to sub-optimal sentence representations. In this paper, we present a Simple method named Self-Contrastive Learning (SSCL) to alleviate this issue, which samples negatives from PLMs intermediate layers, improving the quality of the sentence representation. Our proposed method is quite simple and can be easily extended to various state-of-the-art models for performance boosting, which can be seen as a plug-and-play contrastive framework for learning unsupervised sentence representation. Extensive results prove that SSCL brings the superior performance improvements of different strong baselines (e.g., BERT and SimCSE) on Semantic Textual Similarity and Transfer datasets. Our codes are available at https://github.com/nuochenpku/SSCL.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 26 of its 49 references that are in the map. First posted on arXiv 9 May 2023.

Builds on

26 of 49 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  3. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  4. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  5. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  6. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  7. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  8. 2020A Note on Over-Smoothing for Graph Neural Networkscited by 245
  9. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  10. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 198
  11. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  12. 2022Rethinking Graph Neural Networks for Anomaly Detectioncited by 185
14 more
  1. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  2. 2020An Unsupervised Sentence Embedding Method by Mutual Information Maximizationcited by 91
  3. 2016Learning Distributed Representations of Sentences from Unlabelled Datacited by 90
  4. 2022Revisiting Over-smoothing in BERT from the Perspective of Graphcited by 71
  5. 2021Self-supervised Contrastive Cross-Modality Representation Learning for Spoken Question Answeringcited by 39
  6. 2022SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with Soft Negative Samplescited by 39
  7. 2022End-to-end Spoken Conversational Question Answering: Task, Dataset and Modelcited by 37
  8. 2021Exploring the Role of BERT Token Representations to Explain Sentence Probing Resultscited by 29
  9. 2021Self-supervised Dialogue Learning for Spoken Conversational Question Answeringcited by 27
  10. 2022Semi-Supervised Hierarchical Graph Classificationcited by 15
  11. 2022Handling Missing Data via Max-Entropy Regularized Graph Autoencodercited by 12
  12. 2021From Good to Best: Two-Stage Training for Cross-lingual Machine Reading Comprehensioncited by 10
  13. 2022Bridging the Gap between Language Models and Cross-Lingual Sequence Labelingcited by 7
  14. 2023Bridge the Gap between Language models and Tabular Understandingcited by 5

Built on by

Cited by year

Citing papers per year: 2023: 6; 2024: 5; 2025: 5.

16 papers.

Sort:

By area:

  1. 2024The Hidden Attention of Mamba Modelscited by 88
  2. 2024The Oscars of AI Theater: A Survey on Role-Playing with Language Modelscited by 47
  3. 2024Simple Techniques for Enhancing Sentence Embeddings in Generative Language Modelscited by 26
  4. 2023Is Bigger and Deeper Always Better? Probing LLaMA Across Scales and Layerscited by 20
  5. 2023A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the ChatGPT Era and Beyondcited by 11
  6. 2023Graph Convolutions Enrich the Self-Attention in Transformers!cited by 11
  7. 2023CoT-BERT: Enhancing Unsupervised Sentence Representation through Chain-of-Thoughtcited by 8
  8. 2024Pcc-tuning: Breaking the Contrastive Learning Ceiling in Semantic Textual Similaritycited by 5
  9. 2025CSE-SFP: Enabling Unsupervised Sentence Representation Learning via a Single Forward Passcited by 3
  10. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  11. 2023Unsupervised Sentence Representation Learning with Frequency-induced Adversarial Tuning and Incomplete Sentence Filteringcited by 2
  12. 2024SIBO: A Simple Booster for Parameter-Efficient Fine-Tuningcited by 2
4 more
  1. 2025STEP: Staged Parameter-Efficient Pre-training for Large Language Modelscited by 2
  2. 2025CoDiEmb: A Collaborative yet Distinct Framework for Unified Representation Learning in Information Retrieval and Semantic Textual Similaritycited by 2
  3. 2023EMMA-X: An EM-like Multilingual Pre-training Algorithm for Cross-lingual Representation Learningcited by 1
  4. 2025Text2Token: Unsupervised Text Representation Learning with Token Target Predictioncited by 0