# Reordering Examples Helps during Priming-based Few-Shot Learning

arXiv: https://arxiv.org/abs/2106.01751 · HTML: https://mlsanity.com/paper/2106.01751/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 34 references that are in the map.
- First posted on arXiv 3 Jun 2021.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines](https://mlsanity.com/paper/2006.04884/) (2020)

## Built on by (most cited first, 30 of 50)

- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Augmented Language Models: a Survey](https://mlsanity.com/paper/2302.07842/) (2023)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [Reframing Instructional Prompts to GPTk's Language](https://mlsanity.com/paper/2109.07830/) (2021)
- [Foundation Models for Decision Making: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2303.04129/) (2023)
- [Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Ordering](https://mlsanity.com/paper/2212.10375/) (2022)
- [Prompting PaLM for Translation: Assessing Strategies and Performance](https://mlsanity.com/paper/2211.09102/) (2022)
- [GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Models](https://mlsanity.com/paper/2203.07281/) (2022)
- [Dr.ICL: Demonstration-Retrieved In-context Learning](https://mlsanity.com/paper/2305.14128/) (2023)
- [ProGen: Progressive Zero-shot Dataset Generation via In-context Feedback](https://mlsanity.com/paper/2210.12329/) (2022)
- [Few-Shot Bot: Prompt-Based Learning for Dialogue Systems](https://mlsanity.com/paper/2110.08118/) (2021)
- [Exploring Diverse In-Context Configurations for Image Captioning](https://mlsanity.com/paper/2305.14800/) (2023)
- [In-context Example Selection with Influences](https://mlsanity.com/paper/2302.11042/) (2023)
- [Good Examples Make A Faster Learner: Simple Demonstration-based Learning for Low-resource NER](https://mlsanity.com/paper/2110.08454/) (2021)
- [True Few-Shot Learning with Prompts -- A Real-World Perspective](https://mlsanity.com/paper/2111.13440/) (2021)
- [Decoupling Knowledge from Memorization: Retrieval-augmented Prompt Learning](https://mlsanity.com/paper/2205.14704/) (2022)
- [SOEN-101: Code Generation by Emulating Software Process Models Using Large Language Model Agents](https://mlsanity.com/paper/2403.15852/) (2024)
- [Smoothie: Label Free Language Model Routing](https://mlsanity.com/paper/2412.04692/) (2024)
- [GPT-3-driven pedagogical agents for training children's curious question-asking skills](https://mlsanity.com/paper/2211.14228/) (2022)
- [Co-training Improves Prompt-based Learning for Large Language Models](https://mlsanity.com/paper/2202.00828/) (2022)
- [Multi-Stage Prompting for Knowledgeable Dialogue Generation](https://mlsanity.com/paper/2203.08745/) (2022)
- [Order Matters: Exploring Order Sensitivity in Multimodal Large Language Models](https://mlsanity.com/paper/2410.16983/) (2024)
- [A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomness](https://mlsanity.com/paper/2312.01082/) (2023)
- [Improving Probability-based Prompt Selection Through Unified Evaluation and Analysis](https://mlsanity.com/paper/2305.14877/) (2023)
- [Semantic-Oriented Unlabeled Priming for Large-Scale Language Models](https://mlsanity.com/paper/2202.06133/) (2022)
- [Mimic In-Context Learning for Multimodal Tasks](https://mlsanity.com/paper/2504.08851/) (2025)
- [Ambiguity-Aware In-Context Learning with Large Language Models](https://mlsanity.com/paper/2309.07900/) (2023)
- [Is Relevance Propagated from Retriever to Generator in RAG?](https://mlsanity.com/paper/2502.15025/) (2025)
- [Unlocking the Power of LLM Uncertainty for Active In-Context Example Selection](https://mlsanity.com/paper/2408.09172/) (2024)

## Abstract (arXiv)

The ability to learn from limited data, or few-shot learning, is a desirable and often critical requirement for NLP systems. While many existing methods do poorly at learning from a handful of examples, large pretrained language models have recently been shown to be efficient few-shot learners. One approach to few-shot learning, which does not require finetuning of model parameters, is to augment the language model's input with priming text which is typically constructed using task specific descriptions and examples. In this work, we further explore priming-based few-shot learning, with focus on using examples as prompts. We show that presenting examples in the right order is key for generalization. We introduce PERO (Prompting with Examples in the Right Order), where we formulate few-shot learning as search over the set of permutations of the training examples. We show that PERO can learn to generalize efficiently using as few as 10 examples, in contrast to existing approaches. While the newline token is a natural choice for separating the examples in the prompt, we show that learning a new separator token can potentially provide further gains in performance. We demonstrate the effectiveness of the proposed method on the tasks of sentiment classification, natural language inference and fact retrieval. Finally, we analyze the learned prompts to reveal novel insights, including the idea that two training examples in the right order alone can provide competitive performance for sentiment classification and natural language inference.