# Contrastive Learning for Prompt-Based Few-Shot Language Learners

arXiv: https://arxiv.org/abs/2205.01308 · HTML: https://mlsanity.com/paper/2205.01308/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 28 references that are in the map.
- First posted on arXiv 3 May 2022.

## Builds on (most cited first, 25 of 25)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [Towards Understanding and Mitigating Social Biases in Language Models](https://mlsanity.com/paper/2106.13219/) (2021)
- [Improving and Simplifying Pattern Exploiting Training](https://mlsanity.com/paper/2103.11955/) (2021)
- [Bootstrapping Semantic Segmentation with Regional Contrast](https://mlsanity.com/paper/2104.04465/) (2021)
- [AEDA: An Easier Data Augmentation Technique for Text Classification](https://mlsanity.com/paper/2108.13230/) (2021)
- [LoCo: Local Contrastive Representation Learning](https://mlsanity.com/paper/2008.01342/) (2020)
- [Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Tasks](https://mlsanity.com/paper/2009.08445/) (2020)
- [A Closer Look At Feature Space Data Augmentation For Few-Shot Intent Classification](https://mlsanity.com/paper/1910.04176/) (2019)
- [Few-Shot Text Classification with Triplet Networks, Data Augmentation, and Curriculum Learning](https://mlsanity.com/paper/2103.07552/) (2021)
- [Meta-Learning for Few-Shot NMT Adaptation](https://mlsanity.com/paper/2004.02745/) (2020)
- [Label Hallucination for Few-Shot Classification](https://mlsanity.com/paper/2112.03340/) (2021)
- [Few-shot learning through contextual data augmentation](https://mlsanity.com/paper/2103.16911/) (2021)
- [Semi-Supervised Few-Shot Intent Classification and Slot Filling](https://mlsanity.com/paper/2109.08754/) (2021)
- [MetaPix: Domain Transfer for Semantic Segmentation by Meta Pixel Weighting](https://mlsanity.com/paper/2110.01777/) (2021)

## Built on by (most cited first, 17 of 17)

- [On Large Language Model Continual Unlearning](https://mlsanity.com/paper/2407.10223/) (2024)
- [ConDA: Contrastive Domain Adaptation for AI-generated Text Detection](https://mlsanity.com/paper/2309.03992/) (2023)
- [LM-CPPF: Paraphrasing-Guided Data Augmentation for Contrastive Prompt-Based Few-Shot Fine-Tuning](https://mlsanity.com/paper/2305.18169/) (2023)
- [Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2209.09433/) (2022)
- [Few-shot Multimodal Sentiment Analysis based on Multimodal Probabilistic Fusion Prompts](https://mlsanity.com/paper/2211.06607/) (2022)
- [Prompt-Based Bias Calibration for Better Zero/Few-Shot Learning of Language Models](https://mlsanity.com/paper/2402.10353/) (2024)
- [Emotion-Conditioned Text Generation through Automatic Prompt Optimization](https://mlsanity.com/paper/2308.04857/) (2023)
- [ML-LMCL: Mutual Learning and Large-Margin Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding](https://mlsanity.com/paper/2311.11375/) (2023)
- [Few-shot Personalized Scanpath Prediction](https://mlsanity.com/paper/2504.05499/) (2025)
- [Across-Game Engagement Modelling via Few-Shot Learning](https://mlsanity.com/paper/2409.13002/) (2024)
- [SetCSE: Set Operations using Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2404.17606/) (2024)
- [Let Me Check the Examples: Enhancing Demonstration Learning via Explicit Imitation](https://mlsanity.com/paper/2209.00455/) (2022)
- [Prompt-enhanced Network for Hateful Meme Classification](https://mlsanity.com/paper/2411.07527/) (2024)
- [ConsPrompt: Exploiting Contrastive Samples for Fewshot Prompt Learning](https://mlsanity.com/paper/2211.04118/) (2022)
- [Train Once for All: A Transitional Approach for Efficient Aspect Sentiment Triplet Extraction](https://mlsanity.com/paper/2412.00208/) (2024)
- [Advancing Uto-Aztecan Language Technologies: A Case Study on the Endangered Comanche Language](https://mlsanity.com/paper/2505.18159/) (2025)
- [Rotation Control Unlearning: Quantifying and Controlling Continuous Unlearning for LLM with The Cognitive Rotation Space](https://mlsanity.com/paper/2509.25743/) (2025)

## Abstract (arXiv)

The impressive performance of GPT-3 using natural language prompts and in-context learning has inspired work on better fine-tuning of moderately-sized models under this paradigm. Following this line of work, we present a contrastive learning framework that clusters inputs from the same class for better generality of models trained with only limited examples. Specifically, we propose a supervised contrastive framework that clusters inputs from the same class under different augmented "views" and repel the ones from different classes. We create different "views" of an example by appending it with different language prompts and contextual demonstrations. Combining a contrastive loss with the standard masked language modeling (MLM) loss in prompt-based few-shot learners, the experimental results show that our method can improve over the state-of-the-art methods in a diverse set of 15 language tasks. Our framework makes minimal assumptions on the task or the base model, and can be applied to many recent methods with little modification. The code will be made available at: https://github.com/yiren-jian/LM-SupCon.