# MEAL: Stable and Active Learning for Few-Shot Prompting

arXiv: https://arxiv.org/abs/2211.08358 · HTML: https://mlsanity.com/paper/2211.08358/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 15 Nov 2022.

## Builds on (most cited first, 25 of 25)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds](https://mlsanity.com/paper/1906.03671/) (2019)
- [Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping](https://mlsanity.com/paper/2002.06305/) (2020)
- [On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines](https://mlsanity.com/paper/2006.04884/) (2020)
- [Active Learning by Acquiring Contrastive Examples](https://mlsanity.com/paper/2109.03764/) (2021)
- [Stability](https://mlsanity.com/paper/1310.0150/) (2013)
- [Improving and Simplifying Pattern Exploiting Training](https://mlsanity.com/paper/2103.11955/) (2021)
- [Diverse mini-batch Active Learning](https://mlsanity.com/paper/1901.05954/) (2019)
- [Few-Shot Text Generation with Pattern-Exploiting Training](https://mlsanity.com/paper/2012.11926/) (2020)
- [Revisiting Uncertainty-based Query Strategies for Active Learning with Transformers](https://mlsanity.com/paper/2107.05687/) (2021)
- [A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters](https://mlsanity.com/paper/2012.15682/) (2020)
- [Sampling Bias in Deep Active Classification: An Empirical Study](https://mlsanity.com/paper/1909.09389/) (2019)
- [FewNLU: Benchmarking State-of-the-Art Methods for Few-Shot Natural Language Understanding](https://mlsanity.com/paper/2109.12742/) (2021)
- [Multi-task Active Learning for Pre-trained Transformer-based Models](https://mlsanity.com/paper/2208.05379/) (2022)

## Built on by (most cited first, 20 of 20)

- [Active Prompting with Chain-of-Thought for Large Language Models](https://mlsanity.com/paper/2302.12246/) (2023)
- [Active Learning Principles for In-Context Learning with Large Language Models](https://mlsanity.com/paper/2305.14264/) (2023)
- [The language of prompting: What linguistic properties make a prompt successful?](https://mlsanity.com/paper/2311.01967/) (2023)
- [Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even Performance](https://mlsanity.com/paper/2402.12819/) (2024)
- [Effects of diversity incentives on sample diversity and downstream model performance in LLM-based text augmentation](https://mlsanity.com/paper/2401.06643/) (2024)
- [A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomness](https://mlsanity.com/paper/2312.01082/) (2023)
- [MorphPiece : A Linguistic Tokenizer for Large Language Models](https://mlsanity.com/paper/2307.07262/) (2023)
- [LLMs vs Established Text Augmentation Techniques for Classification: When do the Benefits Outweight the Costs?](https://mlsanity.com/paper/2408.16502/) (2024)
- [Flexible and Adaptable Summarization via Expertise Separation](https://mlsanity.com/paper/2406.05360/) (2024)
- [Use Random Selection for Now: Investigation of Few-Shot Selection Strategies in LLM-based Text Augmentation for Classification](https://mlsanity.com/paper/2410.10756/) (2024)
- [Do prompt positions really matter?](https://mlsanity.com/paper/2305.14493/) (2023)
- [Automatic Combination of Sample Selection Strategies for Few-Shot Learning](https://mlsanity.com/paper/2402.03038/) (2024)
- [StablePT: Towards Stable Prompting for Few-shot Learning via Input Separation](https://mlsanity.com/paper/2404.19335/) (2024)
- [Fighting Randomness with Randomness: Mitigating Optimisation Instability of Fine-Tuning using Delayed Ensemble and Noisy Interpolation](https://mlsanity.com/paper/2406.12471/) (2024)
- [PromptAL: Sample-Aware Dynamic Soft Prompts for Few-Shot Active Learning](https://mlsanity.com/paper/2507.16424/) (2025)
- [Sociocultural knowledge is needed for selection of shots in hate speech detection tasks](https://mlsanity.com/paper/2304.01890/) (2023)
- [On Sensitivity of Learning with Limited Labelled Data to the Effects of Randomness: Impact of Interactions and Systematic Choices](https://mlsanity.com/paper/2402.12817/) (2024)
- [Large Language Models are Demonstration Pre-Selectors for Themselves](https://mlsanity.com/paper/2506.06033/) (2025)
- [The Power of Adaptation: Boosting In-Context Learning through Adaptive Prompting](https://mlsanity.com/paper/2412.17891/) (2024)
- [Active Few-Shot Learning for Vertex Classification Starting from an Unlabeled Dataset](https://mlsanity.com/paper/2504.18696/) (2025)

## Abstract (arXiv)

Few-shot classification has made great strides due to foundation models that, through priming and prompting, are highly effective few-shot learners. However, this approach has high variance both across different sets of few shots (data selection) and across different finetuning runs (run variability). This is problematic not only because it impedes the fair comparison of different approaches, but especially because it makes few-shot learning too unreliable for many real-world applications. To alleviate these issues, we make two contributions for more stable and effective few-shot learning: First, we propose novel ensembling methods and show that they substantially reduce run variability. Second, we introduce a new active learning (AL) criterion for data selection and present the first AL-based approach specifically tailored towards prompt-based learning. In our experiments, we show that our combined method, MEAL (Multiprompt finetuning and prediction Ensembling with Active Learning), improves overall performance of prompt-based finetuning by 2.3 points on five diverse tasks. We publicly share our code and data splits in https://github.com/akoksal/MEAL.