mlsanityData to 7 Oct 2026

Batch-Softmax Contrastive Loss for Pairwise Sentence Scoring Tasks

arXiv 2110.15725 ↗First posted 10 Oct 2021cs.CL cs.AI cs.IRcited by 2

Not on the map. Why?

Abstract (arXiv)

The use of contrastive loss for representation learning has become prominent in computer vision, and it is now getting attention in Natural Language Processing (NLP). Here, we explore the idea of using a batch-softmax contrastive loss when fine-tuning large-scale pre-trained transformer models to learn better task-specific sentence embeddings for pairwise sentence scoring tasks. We introduce and study a number of variations in the calculation of the loss as well as in the overall training procedure; in particular, we find that data shuffling can be quite important. Our experimental results show sizable improvements on a number of datasets and pairwise sentence scoring tasks including classification, ranking, and regression. Finally, we offer detailed analysis and discussion, which should be useful for researchers aiming to explore the utility of contrastive loss in NLP.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 25 of its 29 references that are in the map. First posted on arXiv 10 Oct 2021.

Builds on

25 of 29 references.

References · 25

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2020Supervised Contrastive Learningcited by 2,803
  5. 2017Billion-scale similarity search with GPUscited by 2,477
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  7. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  8. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  9. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  10. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  11. 2018MINE: Mutual Information Neural Estimationcited by 680
  12. 2019On Mutual Information Maximization for Representation Learningcited by 315
13 more
  1. 2020Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningcited by 275
  2. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  3. 2017Efficient Natural Language Response Suggestion for Smart Replycited by 228
  4. 2018An efficient framework for learning sentence representationscited by 197
  5. 2020CERT: Contrastive Self-supervised Learning for Language Understandingcited by 169
  6. 2020Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Taskscited by 136
  7. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  8. 2020That is a Known Lie: Detecting Previously Fact-Checked Claimscited by 82
  9. 2017On Sampling Strategies for Neural Network-based Collaborative Filteringcited by 71
  10. 2018Learning Semantic Textual Similarity from Conversationscited by 41
  11. 2019ANTIQUE: A Non-Factoid Question Answering Benchmarkcited by 24
  12. 2019SemEval-2017 Task 3: Community Question Answeringcited by 13
  13. 2020CAPT: Contrastive Pre-Training for Learning Denoised Sequence Representationscited by 8

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1.

2 papers.

  1. 2022CrowdChecked: Detecting Previously Fact-Checked Claims in Social Mediacited by 9
  2. 2023Instances and Labels: Hierarchy-aware Joint Supervised Contrastive Learning for Hierarchical Multi-Label Text Classificationcited by 4