mlsanityData to 14 Oct 2026

CSE-SFP: Enabling Unsupervised Sentence Representation Learning via a Single Forward Pass

arXiv 2505.00389 ↗First posted 1 May 2025cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

As a fundamental task in Information Retrieval and Computational Linguistics, sentence representation has profound implications for a wide range of practical applications such as text clustering, content analysis, question-answering systems, and web search. Recent advances in pre-trained language models (PLMs) have driven remarkable progress in this field, particularly through unsupervised embedding derivation methods centered on discriminative PLMs like BERT. However, due to time and computational constraints, few efforts have attempted to integrate unsupervised sentence representation with generative PLMs, which typically possess much larger parameter sizes. Given that state-of-the-art models in both academia and industry are predominantly based on generative architectures, there is a pressing need for an efficient unsupervised text representation framework tailored to decoder-only PLMs. To address this concern, we propose CSE-SFP, an innovative method that exploits the structural characteristics of generative models. Compared to existing strategies, CSE-SFP requires only a single forward pass to perform effective unsupervised contrastive learning. Rigorous experimentation demonstrates that CSE-SFP not only produces higher-quality embeddings but also significantly reduces both training time and memory consumption. Furthermore, we introduce two ratio metrics that jointly assess alignment and uniformity, thereby providing a more robust means for evaluating the semantic spatial properties of encoding models.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 35 of its 45 references that are in the map. First posted on arXiv 1 May 2025.

Builds on

35 of 45 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  11. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  12. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
23 more
  1. 2019BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformercited by 964
  2. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  3. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  4. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
  5. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 354
  6. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  7. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  8. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 132
  9. 2023Scaling Sentence Embeddings with Large Language Modelscited by 85
  10. 2020On the Sentence Embeddings from Pre-trained Language Modelscited by 83
  11. 2022PromptBERT: Improving BERT Sentence Embeddings with Promptscited by 75
  12. 2021ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embeddingcited by 74
  13. 2022Revisiting Over-smoothing in BERT from the Perspective of Graphcited by 72
  14. 2022SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with Soft Negative Samplescited by 39
  15. 2024Simple Techniques for Enhancing Sentence Embeddings in Generative Language Modelscited by 26
  16. 2023RankCSE: Unsupervised Sentence Representations Learning via Learning to Rankcited by 23
  17. 2022Improving Contrastive Learning of Sentence Embeddings with Case-Augmented Positives and Retrieved Negativescited by 16
  18. 2023Alleviating Over-smoothing for Unsupervised Sentence Representationcited by 16
  19. 2022A Sentence is Worth 128 Pseudo Tokens: A Semantic-Aware Contrastive Learning Framework for Sentence Embeddingscited by 9
  20. 2022Differentiable Data Augmentation for Contrastive Sentence Representation Learningcited by 9
  21. 2023CoT-BERT: Enhancing Unsupervised Sentence Representation through Chain-of-Thoughtcited by 8
  22. 2024Advancing Semantic Textual Similarity Modeling: A Regression Framework with Translated ReLU and Smooth K2 Losscited by 5
  23. 2024Pcc-tuning: Breaking the Contrastive Learning Ceiling in Semantic Textual Similaritycited by 5

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  2. 2025CoDiEmb: A Collaborative yet Distinct Framework for Unified Representation Learning in Information Retrieval and Semantic Textual Similaritycited by 2
  3. 2025PoLi-RL: A Point-to-List Reinforcement Learning Framework for Conditional Semantic Textual Similaritycited by 1