mlsanityData to 7 Oct 2026

Unveiling Key Aspects of Fine-Tuning in Sentence Embeddings: A Representation Rank Analysis

arXiv 2405.11297 ↗First posted 18 May 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

The latest advancements in unsupervised learning of sentence embeddings predominantly involve employing contrastive learning-based (CL-based) fine-tuning over pre-trained language models. In this study, we analyze the latest sentence embedding methods by adopting representation rank as the primary tool of analysis. We first define Phase 1 and Phase 2 of fine-tuning based on when representation rank peaks. Utilizing these phases, we conduct a thorough analysis and obtain essential findings across key aspects, including alignment and uniformity, linguistic abilities, and correlation between performance and rank. For instance, we find that the dynamics of the key aspects can undergo significant changes as fine-tuning transitions from Phase 1 to Phase 2. Based on these findings, we experiment with a rank reduction (RR) strategy that facilitates rapid and stable fine-tuning of the latest CL-based methods. Through empirical investigations, we showcase the efficacy of RR in enhancing the performance and stability of five state-of-the-art sentence embedding methods.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 36 references that are in the map. First posted on arXiv 18 May 2024.

Builds on

24 of 36 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  6. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  7. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  8. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  9. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  10. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  11. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  12. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 145
12 more
  1. 2021Decoupled Contrastive Learningcited by 141
  2. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 131
  3. 2022PromptBERT: Improving BERT Sentence Embeddings with Promptscited by 75
  4. 2021ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embeddingcited by 73
  5. 2023Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanismcited by 32
  6. 2022Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learningcited by 30
  7. 2022PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddingscited by 29
  8. 2022InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddingscited by 25
  9. 2024Discovering Low-rank Subspaces for Language-agnostic Multilingual Representationscited by 18
  10. 2023VNE: An Effective Method for Improving Deep Representation by Manipulating Eigenvalue Distributioncited by 17
  11. 2023SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectivescited by 15
  12. 2022miCSE: Mutual Information Contrastive Learning for Low-shot Sentence Embeddingscited by 7

Built on by

Not cited yet.