Estimating class separability of text embeddings with persistent homology
Not on the map. Why?
Abstract (arXiv)
This paper introduces an unsupervised method to estimate the class separability of text datasets from a topological point of view. Using persistent homology, we demonstrate how tracking the evolution of embedding manifolds during training can inform about class separability. More specifically, we show how this technique can be applied to detect when the training process stops improving the separability of the embeddings. Our results, validated across binary and multi-class text classification tasks, show that the proposed method's estimates of class separability align with those obtained from supervised methods. This approach offers a novel perspective on monitoring and improving the fine-tuning of sentence transformers for classification tasks, particularly in scenarios where labeled data is scarce. We also discuss how tracking these quantities can provide additional insights into the properties of the trained classifier.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 21 of its 44 references that are in the map. First posted on arXiv 24 May 2023.
Builds on
21 of 44 references.
References · 21
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
- 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
- 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
- 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
- 2018MINE: Mutual Information Neural Estimationcited by 680
- 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
- 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
- 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 145
- 2022Efficient Few-Shot Learning Without Promptscited by 135
- 2015A roadmap for the computation of persistent homologycited by 126
9 more
- 2019Ripser: efficient computation of Vietoris-Rips persistence barcodescited by 115
- 2020Topology of deep neural networkscited by 111
- 2018Neural Persistence: A Complexity Measure for Deep Neural Networks Using Algebraic Topologycited by 73
- 2021Representation Topology Divergence: A Method for Comparing Neural Network Representationscited by 66
- 2021Activation Landscapes as a Topological Summary of Neural Network Performancecited by 17
- 2021A Novel Intrinsic Measure of Data Separabilitycited by 9
- 2021Characterizing and Measuring the Similarity of Neural Networks with Persistent Homologycited by 8
- 2021Topological Deep Learning: Classification Neural Networkscited by 5
- 2021Persistent Homology Captures the Generalization of Neural Networks Without A Validation Setcited by 4
Built on by
Citing papers per year: 2024: 1.
1 paper.