mlsanityData to 7 Oct 2026

Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning

arXiv 2011.01403 ↗First posted 3 Nov 2020cs.CL cs.LGcited by 275

Not on the map. Why?

Abstract (arXiv)

State-of-the-art natural language understanding classification models follow two-stages: pre-training a large language model on an auxiliary task, and then fine-tuning the model on a task-specific labeled dataset using cross-entropy loss. However, the cross-entropy loss has several shortcomings that can lead to sub-optimal generalization and instability. Driven by the intuition that good generalization requires capturing the similarity between examples in one class and contrasting them with examples in other classes, we propose a supervised contrastive learning (SCL) objective for the fine-tuning stage. Combined with cross-entropy, our proposed SCL loss obtains significant improvements over a strong RoBERTa-Large baseline on multiple datasets of the GLUE benchmark in few-shot learning settings, without requiring specialized architecture, data augmentations, memory banks, or additional unsupervised data. Our proposed fine-tuning objective leads to models that are more robust to different levels of noise in the fine-tuning training data, and can generalize better to related tasks with limited labeled data.

Key facts. Cited by 275 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 58; 2022: 65; 2023: 49; 2024: 53; 2025: 34; 2026: 16.

275 papers.

Sort:

By area:

  1. 2022Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlationscited by 171
  2. 2021Active Learning by Acquiring Contrastive Examplescited by 154
  3. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
  4. 2021Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learningcited by 140
  5. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  6. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  7. 2021Entailment as Few-Shot Learnercited by 122
  8. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  9. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  10. 2022WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domaincited by 117
  11. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  12. 2022Intent Contrastive Learning for Sequential Recommendationcited by 90
38 more
  1. 2022SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Modelscited by 90
  2. 2021Graph-MLP: Node Classification without Message Passing in Graphcited by 73
  3. 2021Contrastive Out-of-Distribution Detection for Pretrained Transformerscited by 70
  4. 2021Category Contrast for Unsupervised Domain Adaptation in Visual Taskscited by 65
  5. 2022Expanding Small-Scale Datasets with Guided Imaginationcited by 61
  6. 2024DeTeCtive: Detecting AI-generated Text via Multi-Level Contrastive Learningcited by 52
  7. 2021Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuningcited by 50
  8. 2022Contrastive Positive Sample Propagation along the Audio-Visual Event Linecited by 50
  9. 2021Few-Shot Intent Detection via Contrastive Pre-Training and Fine-Tuningcited by 46
  10. 2021CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuningcited by 44
  11. 2021Not All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text Classificationcited by 43
  12. 2024Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wildcited by 43
  13. 2021Modeling Discriminative Representations for Out-of-Domain Detection with Supervised Contrastive Learningcited by 42
  14. 2021A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectivescited by 41
  15. 2021Improved Text Classification via Contrastive Adversarial Trainingcited by 38
  16. 2021Contrast and Generation Make BART a Good Dialogue Emotion Recognizercited by 35
  17. 2021Contrastive Learning for Fair Representationscited by 34
  18. 2022Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guaranteescited by 34
  19. 2021STraTA: Self-Training with Task Augmentation for Better Few-shot Learningcited by 33
  20. 2022CoCo: Coherence-Enhanced Machine-Generated Text Detection Under Data Limitation With Contrastive Learningcited by 32
  21. 2023A Survey on Out-of-Distribution Detection in NLPcited by 31
  22. 2021Noise Stability Regularization for Improving BERT Fine-tuningcited by 29
  23. 2023Is Fine-tuning Needed? Pre-trained Language Models Are Near Perfect for Out-of-Domain Detectioncited by 29
  24. 2023NADI 2023: The Fourth Nuanced Arabic Dialect Identification Shared Taskcited by 29
  25. 2022New Intent Discovery with Pre-training and Contrastive Learningcited by 28
  26. 2022On the Impossible Safety of Large AI Modelscited by 27
  27. 2024Heterogeneous Contrastive Learning for Foundation Models and Beyondcited by 27
  28. 2021ConvFiT: Conversational Fine-Tuning of Pretrained Language Modelscited by 25
  29. 2021Exploring Task Difficulty for Few-Shot Relation Extractioncited by 24
  30. 2021Topic-Aware Contrastive Learning for Abstractive Dialogue Summarizationcited by 23
  31. 2022FedClassAvg: Local Representation Learning for Personalized Federated Learning on Heterogeneous Neural Networkscited by 23
  32. 2021Revisiting Self-Training for Few-Shot Learning of Language Modelcited by 22
  33. 2022Dual Contrastive Learning: Text Classification via Label-Aware Data Augmentationcited by 22
  34. 2022Neighborhood Collective Estimation for Noisy Label Identification and Correctioncited by 22
  35. 2023Facilitating Contrastive Learning of Discourse Relational Senses by Exploiting the Hierarchy of Sense Relationscited by 22
  36. 2022Online Continual Learning with Contrastive Vision Transformercited by 21
  37. 2025Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignmentcited by 20
  38. 2022Superficial White Matter Analysis: An Efficient Point-cloud-based Deep Learning Framework with Supervised Contrastive Learning for Consistent Tractography Parcellation across Populations and dMRI Acquisitionscited by 19

50 most cited of 275 All 275 (JSON)