HiCL: Hierarchical Contrastive Learning of Unsupervised Sentence Embeddings
Not on the map. Why?
Abstract (arXiv)
In this paper, we propose a hierarchical contrastive learning framework, HiCL, which considers local segment-level and global sequence-level relationships to improve training efficiency and effectiveness. Traditional methods typically encode a sequence in its entirety for contrast with others, often neglecting local representation learning, leading to challenges in generalizing to shorter texts. Conversely, HiCL improves its effectiveness by dividing the sequence into several segments and employing both local and global contrastive learning to model segment-level and sequence-level relationships. Further, considering the quadratic time complexity of transformers over input tokens, HiCL boosts training efficiency by first encoding short segments and then aggregating them to obtain the sequence representation. Extensive experiments show that HiCL enhances the prior top-performing SNCSE model across seven extensively evaluated STS tasks, with an average increase of +0.2% observed on BERT-large and +0.44% on RoBERTa-large.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 32 of its 49 references that are in the map. First posted on arXiv 15 Oct 2023.
Builds on
32 of 49 references.
References · 32
- 2017Attention Is All You Needcited by 59,932
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
- 2020Exploring Simple Siamese Representation Learningcited by 2,853
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
- 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
- 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
- 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
- 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
20 more
- 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
- 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
- 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
- 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
- 2020Contrastive Learning with Hard Negative Samplescited by 619
- 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
- 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
- 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
- 2020CERT: Contrastive Self-supervised Learning for Language Understandingcited by 169
- 2019Hierarchical Transformers for Multi-Document Summarizationcited by 132
- 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 131
- 2019HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarizationcited by 120
- 2021Hierarchical Transformers Are More Efficient Language Modelscited by 99
- 2022PromptBERT: Improving BERT Sentence Embeddings with Promptscited by 75
- 2021ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embeddingcited by 73
- 2022Debiased Contrastive Learning of Unsupervised Sentence Representationscited by 56
- 2022IDPG: An Instance-Dependent Prompt Generation Methodcited by 43
- 2022SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with Soft Negative Samplescited by 39
- 2020Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matchingcited by 35
- 2022InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddingscited by 25
Built on by
Citing papers per year: 2023: 1.
1 paper.