# PERFECT: Prompt-free and Efficient Few-shot Learning with Language Models

arXiv: https://arxiv.org/abs/2204.01172 · HTML: https://mlsanity.com/paper/2204.01172/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 67 references that are in the map.
- First posted on arXiv 3 Apr 2022.

## Builds on (most cited first, 30 of 46)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [Compacter: Efficient Low-Rank Hypercomplex Adapter Layers](https://mlsanity.com/paper/2106.04647/) (2021)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping](https://mlsanity.com/paper/2002.06305/) (2020)
- [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://mlsanity.com/paper/2104.06599/) (2021)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [Measuring the Intrinsic Dimension of Objective Landscapes](https://mlsanity.com/paper/1804.08838/) (2018)

## Built on by (most cited first, 30 of 40)

- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [Zero-Shot Video Question Answering via Frozen Bidirectional Language Models](https://mlsanity.com/paper/2206.08155/) (2022)
- [Personality Traits in Large Language Models](https://mlsanity.com/paper/2307.00184/) (2023)
- [Efficient Few-Shot Learning Without Prompts](https://mlsanity.com/paper/2209.11055/) (2022)
- [Efficient Methods for Natural Language Processing: A Survey](https://mlsanity.com/paper/2209.00099/) (2022)
- [Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU](https://mlsanity.com/paper/2310.04928/) (2023)
- [Quantifying Context Mixing in Transformers](https://mlsanity.com/paper/2301.12971/) (2023)
- [Large Language Models can Implement Policy Iteration](https://mlsanity.com/paper/2210.03821/) (2022)
- [Improving Few-Shot Performance of Language Models via Nearest Neighbor Calibration](https://mlsanity.com/paper/2212.02216/) (2022)
- [On the Domain Adaptation and Generalization of Pretrained Language Models: A Survey](https://mlsanity.com/paper/2211.03154/) (2022)
- [BLIND: Bias Removal With No Demographics](https://mlsanity.com/paper/2212.10563/) (2022)
- [Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even Performance](https://mlsanity.com/paper/2402.12819/) (2024)
- [ActiveLLM: Large Language Model-based Active Learning for Textual Few-Shot Scenarios](https://mlsanity.com/paper/2405.10808/) (2024)
- [Improving the Sample Efficiency of Prompt Tuning with Domain Adaptation](https://mlsanity.com/paper/2210.02952/) (2022)
- [Self-supervised Meta-Prompt Learning with Meta-Gradient Regularization for Few-shot Generalization](https://mlsanity.com/paper/2303.12314/) (2023)
- [Question-Analysis Prompting Improves LLM Performance in Reasoning Tasks](https://mlsanity.com/paper/2407.03624/) (2024)
- [A Universal Discriminator for Zero-Shot Generalization](https://mlsanity.com/paper/2211.08099/) (2022)
- [Scalable Prompt Generation for Semi-supervised Learning with Language Models](https://mlsanity.com/paper/2302.09236/) (2023)
- [Can Foundation Models Help Us Achieve Perfect Secrecy?](https://mlsanity.com/paper/2205.13722/) (2022)
- [Automated Few-shot Classification with Instruction-Finetuned Language Models](https://mlsanity.com/paper/2305.12576/) (2023)
- [Active Few-Shot Learning for Text Classification](https://mlsanity.com/paper/2502.18782/) (2025)
- [Embedding Recycling for Language Models](https://mlsanity.com/paper/2207.04993/) (2022)
- [KnowPrefix-Tuning: A Two-Stage Prefix-Tuning Framework for Knowledge-Grounded Dialogue Generation](https://mlsanity.com/paper/2306.15430/) (2023)
- [Zero-Shot and Few-Shot Video Question Answering with Multi-Modal Prompts](https://mlsanity.com/paper/2309.15915/) (2023)
- [Multi-Level Fine-Tuning, Data Augmentation, and Few-Shot Learning for Specialized Cyber Threat Intelligence](https://mlsanity.com/paper/2207.11076/) (2022)
- [Transductive Learning for Textual Few-Shot Classification in API-based Embedding Models](https://mlsanity.com/paper/2310.13998/) (2023)
- [Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guide](https://mlsanity.com/paper/2411.09539/) (2024)
- [AdaSent: Efficient Domain-Adapted Sentence Embeddings for Few-Shot Classification](https://mlsanity.com/paper/2311.00408/) (2023)
- [Efficient Few-shot Learning for Multi-label Classification of Scientific Documents with Many Classes](https://mlsanity.com/paper/2410.05770/) (2024)
- [Conformal Predictor for Improving Zero-shot Text Classification Efficiency](https://mlsanity.com/paper/2210.12619/) (2022)

## Abstract (arXiv)

Current methods for few-shot fine-tuning of pretrained masked language models (PLMs) require carefully engineered prompts and verbalizers for each new task to convert examples into a cloze-format that the PLM can score. In this work, we propose PERFECT, a simple and efficient method for few-shot fine-tuning of PLMs without relying on any such handcrafting, which is highly effective given as few as 32 data points. PERFECT makes two key design choices: First, we show that manually engineered task prompts can be replaced with task-specific adapters that enable sample-efficient fine-tuning and reduce memory and storage costs by roughly factors of 5 and 100, respectively. Second, instead of using handcrafted verbalizers, we learn new multi-token label embeddings during fine-tuning, which are not tied to the model vocabulary and which allow us to avoid complex auto-regressive decoding. These embeddings are not only learnable from limited data but also enable nearly 100x faster training and inference. Experiments on a wide range of few-shot NLP tasks demonstrate that PERFECT, while being simple and efficient, also outperforms existing state-of-the-art few-shot learning methods. Our code is publicly available at https://github.com/facebookresearch/perfect.git.