mlsanityData to 14 Oct 2026

miCSE: Mutual Information Contrastive Learning for Low-shot Sentence Embeddings

arXiv 2211.04928 ↗First posted 9 Nov 2022cs.CL cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

This paper presents miCSE, a mutual information-based contrastive learning framework that significantly advances the state-of-the-art in few-shot sentence embedding. The proposed approach imposes alignment between the attention pattern of different views during contrastive learning. Learning sentence embeddings with miCSE entails enforcing the structural consistency across augmented views for every sentence, making contrastive self-supervised learning more sample efficient. As a result, the proposed approach shows strong performance in the few-shot learning domain. While it achieves superior results compared to state-of-the-art methods on multiple benchmarks in few-shot learning, it is comparable in the full-shot scenario. This study opens up avenues for efficient self-supervised learning methods that are more robust than current contrastive methods for sentence embedding.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 45 of its 66 references that are in the map. First posted on arXiv 9 Nov 2022.

Builds on

45 of 66 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  7. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  8. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  9. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  10. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 807
  11. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  12. 2020AdapterHub: A Framework for Adapting Transformerscited by 444
33 more
  1. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  2. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 255
  3. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  4. 2020MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classificationcited by 187
  5. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 132
  6. 2019Augmenting Data with Mixup for Sentence Classification: An Empirical Studycited by 125
  7. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  8. 2020An Unsupervised Sentence Embedding Method by Mutual Information Maximizationcited by 91
  9. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  10. 2017Data Noising as Smoothing in Neural Network Language Modelscited by 78
  11. 2021ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embeddingcited by 74
  12. 2020AdvAug: Robust Adversarial Augmentation for Neural Machine Translationcited by 67
  13. 2022Debiased Contrastive Learning of Unsupervised Sentence Representationscited by 56
  14. 2020On Mutual Information in Contrastive Learning for Visual Representationscited by 54
  15. 2020Bayesian Attention Modulescited by 51
  16. 2021Artificial Text Detection via Examining the Topology of Attention Mapscited by 49
  17. 2014Efficient Estimation of Mutual Information for Strongly Dependent Variablescited by 46
  18. 2021Mutual Contrastive Learning for Visual Representation Learningcited by 42
  19. 2018Text Data Augmentation Made Simple By Leveraging NLP Cloud APIscited by 40
  20. 2018Latent Alignment and Variational Attentioncited by 36
  21. 2017Variational Attention for Sequence-to-Sequence Modelscited by 29
  22. 2022PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddingscited by 29
  23. 2022InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddingscited by 26
  24. 2021Decomposed Mutual Information Estimation for Contrastive Representation Learningcited by 25
  25. 2022Exploring the Impact of Negative Samples of Contrastive Learning: A Case Study of Sentence Embeddingcited by 25
  26. 2021Attention Can Reflect Syntactic Structure (If You Let It)cited by 22
  27. 2021Virtual Augmentation Supported Contrastive Learning of Sentence Representationscited by 21
  28. 2021Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillationscited by 20
  29. 2022SCD: Self-Contrastive Decorrelation for Sentence Embeddingscited by 20
  30. 2022Improving Contrastive Learning of Sentence Embeddings with Case-Augmented Positives and Retrieved Negativescited by 16
  31. 2022The Topological BERT: Transforming Attention into Topology for Natural Language Processingcited by 10
  32. 2020Structural Supervision Improves Few-Shot Learning and Syntactic Generalization in Neural Language Modelscited by 8
  33. 2022An Information Minimization Based Contrastive Learning Model for Unsupervised Sentence Embeddings Learningcited by 6

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 2; 2025: 2.

7 papers.

  1. 2025A Survey on Latent Reasoningcited by 81
  2. 2023Headless Language Models: Learning without Predicting with Contrastive Weight Tyingcited by 5
  3. 2023Instance Smoothed Contrastive Learning for Unsupervised Sentence Embeddingcited by 3
  4. 2023Efficient Document Embeddings via Self-Contrastive Bregman Divergence Learningcited by 3
  5. 2024Unveiling Key Aspects of Fine-Tuning in Sentence Embeddings: A Representation Rank Analysiscited by 0
  6. 2024reCSE: Portable Reshaping Features for Sentence Embedding in Self-supervised Contrastive Learningcited by 0
  7. 2025Mutual Information guided Visual Contrastive Learningcited by 0