# Efficient Few-Shot Learning Without Prompts

arXiv: https://arxiv.org/abs/2209.11055 · HTML: https://mlsanity.com/paper/2209.11055/

## Key facts

- Cited by 137 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 35 references that are in the map.
- First posted on arXiv 22 Sep 2022.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping](https://mlsanity.com/paper/2002.06305/) (2020)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://mlsanity.com/paper/2202.01279/) (2022)
- [Revisiting Few-sample BERT Fine-tuning](https://mlsanity.com/paper/2006.05987/) (2020)
- [The Multilingual Amazon Reviews Corpus](https://mlsanity.com/paper/2010.02573/) (2020)
- [Improving and Simplifying Pattern Exploiting Training](https://mlsanity.com/paper/2103.11955/) (2021)
- [I Wish I Would Have Loved This One, But I Didn't -- A Multilingual Dataset for Counterfactual Detection in Product Reviews](https://mlsanity.com/paper/2104.06893/) (2021)
- [RAFT: A Real-World Few-Shot Text Classification Benchmark](https://mlsanity.com/paper/2109.14076/) (2021)
- [PERFECT: Prompt-free and Efficient Few-shot Learning with Language Models](https://mlsanity.com/paper/2204.01172/) (2022)
- [True Few-Shot Learning with Prompts -- A Real-World Perspective](https://mlsanity.com/paper/2111.13440/) (2021)
- [Evaluation of sentence embeddings in downstream and linguistic probing tasks](https://mlsanity.com/paper/1806.06259/) (2018)

## Built on by (most cited first, 30 of 137)

- [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference](https://mlsanity.com/paper/2412.13663/) (2024)
- [Bias Testing and Mitigation in LLM-based Code Generation](https://mlsanity.com/paper/2309.14345/) (2023)
- [Making LLMs Worth Every Penny: Resource-Limited Text Classification in Banking](https://mlsanity.com/paper/2311.06102/) (2023)
- [In-Context Learning for Text Classification with Many Labels](https://mlsanity.com/paper/2309.10954/) (2023)
- [MasakhaNEWS: News Topic Classification for African languages](https://mlsanity.com/paper/2304.09972/) (2023)
- [Embedding-based classifiers can detect prompt injection attacks](https://mlsanity.com/paper/2410.22284/) (2024)
- [PromptMix: A Class Boundary Augmentation Method for Large Language Model Distillation](https://mlsanity.com/paper/2310.14192/) (2023)
- [Intent Detection in the Age of LLMs](https://mlsanity.com/paper/2410.01627/) (2024)
- [QuanTemp: A real-world open-domain benchmark for fact-checking numerical claims](https://mlsanity.com/paper/2403.17169/) (2024)
- [ActiveLLM: Large Language Model-based Active Learning for Textual Few-Shot Scenarios](https://mlsanity.com/paper/2405.10808/) (2024)
- [GLiClass: Generalist Lightweight Model for Sequence Classification Tasks](https://mlsanity.com/paper/2508.07662/) (2025)
- [A Large Language Model Approach to Educational Survey Feedback Analysis](https://mlsanity.com/paper/2309.17447/) (2023)
- [Spam-T5: Benchmarking Large Language Models for Few-Shot Email Spam Detection](https://mlsanity.com/paper/2304.01238/) (2023)
- [Interpretable Medical Diagnostics with Structured Data Extraction by Large Language Models](https://mlsanity.com/paper/2306.05052/) (2023)
- [SynerGPT: In-Context Learning for Personalized Drug Synergy Prediction and Drug Design](https://mlsanity.com/paper/2307.11694/) (2023)
- [A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the ChatGPT Era and Beyond](https://mlsanity.com/paper/2305.12641/) (2023)
- [Breaking the Bank with ChatGPT: Few-Shot Text Classification for Finance](https://mlsanity.com/paper/2308.14634/) (2023)
- [AfriHate: A Multilingual Collection of Hate Speech and Abusive Language Datasets for African Languages](https://mlsanity.com/paper/2501.08284/) (2025)
- [Synthetically generated text for supervised text analysis](https://mlsanity.com/paper/2303.16028/) (2023)
- [MoVer: Motion Verification for Motion Graphics Animations](https://mlsanity.com/paper/2502.13372/) (2025)
- [OCD: Learning to Overfit with Conditional Diffusion Models](https://mlsanity.com/paper/2210.00471/) (2022)
- [Automated Few-shot Classification with Instruction-Finetuned Language Models](https://mlsanity.com/paper/2305.12576/) (2023)
- [Code Red! On the Harmfulness of Applying Off-the-shelf Large Language Models to Programming Tasks](https://mlsanity.com/paper/2504.01850/) (2025)
- [STACC: Code Comment Classification using SentenceTransformers](https://mlsanity.com/paper/2302.13149/) (2023)
- [QAID: Question Answering Inspired Few-shot Intent Detection](https://mlsanity.com/paper/2303.01593/) (2023)
- [NormMark: A Weakly Supervised Markov Model for Socio-cultural Norm Discovery](https://mlsanity.com/paper/2305.16598/) (2023)
- [ILLUMINER: Instruction-tuned Large Language Models as Few-shot Intent Classifier and Slot Filler](https://mlsanity.com/paper/2403.17536/) (2024)
- [Self-Improving Customer Review Response Generation Based on LLMs](https://mlsanity.com/paper/2405.03845/) (2024)
- [Automatic Labelling with Open-source LLMs using Dynamic Label Schema Integration](https://mlsanity.com/paper/2501.12332/) (2025)
- [Fine-Tuning Causal LLMs for Text Classification: Embedding-Based vs. Instruction-Based Approaches](https://mlsanity.com/paper/2512.12677/) (2025)

## Abstract (arXiv)

Recent few-shot methods, such as parameter-efficient fine-tuning (PEFT) and pattern exploiting training (PET), have achieved impressive results in label-scarce settings. However, they are difficult to employ since they are subject to high variability from manually crafted prompts, and typically require billion-parameter language models to achieve high accuracy. To address these shortcomings, we propose SetFit (Sentence Transformer Fine-tuning), an efficient and prompt-free framework for few-shot fine-tuning of Sentence Transformers (ST). SetFit works by first fine-tuning a pretrained ST on a small number of text pairs, in a contrastive Siamese manner. The resulting model is then used to generate rich text embeddings, which are used to train a classification head. This simple framework requires no prompts or verbalizers, and achieves high accuracy with orders of magnitude less parameters than existing techniques. Our experiments show that SetFit obtains comparable results with PEFT and PET techniques, while being an order of magnitude faster to train. We also show that SetFit can be applied in multilingual settings by simply switching the ST body. Our code is available at https://github.com/huggingface/setfit and our datasets at https://huggingface.co/setfit .