# Few-Shot Text Generation with Pattern-Exploiting Training

arXiv: https://arxiv.org/abs/2012.11926 · HTML: https://mlsanity.com/paper/2012.11926/

## Key facts

- Cited by 87 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Dec 2020.

## Built on by (most cited first, 30 of 87)

- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [Promptagator: Few-shot Dense Retrieval From 8 Examples](https://mlsanity.com/paper/2209.11755/) (2022)
- [Generating Training Data with Language Models: Towards Zero-Shot Language Understanding](https://mlsanity.com/paper/2202.04538/) (2022)
- [Reframing Instructional Prompts to GPTk's Language](https://mlsanity.com/paper/2109.07830/) (2021)
- [ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding](https://mlsanity.com/paper/2305.14196/) (2023)
- [BRIO: Bringing Order to Abstractive Summarization](https://mlsanity.com/paper/2203.16804/) (2022)
- [GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Models](https://mlsanity.com/paper/2203.07281/) (2022)
- [Evaluating and Inducing Personality in Pre-trained Language Models](https://mlsanity.com/paper/2206.07550/) (2022)
- [Task Contamination: Language Models May Not Be Few-Shot Anymore](https://mlsanity.com/paper/2312.16337/) (2023)
- [PEER: A Collaborative Language Model](https://mlsanity.com/paper/2208.11663/) (2022)
- [Differentially Private Language Models for Secure Data Sharing](https://mlsanity.com/paper/2210.13918/) (2022)
- [Graph Prompt Learning: A Comprehensive Survey and Beyond](https://mlsanity.com/paper/2311.16534/) (2023)
- [Improving In-Context Few-Shot Learning via Self-Supervised Training](https://mlsanity.com/paper/2205.01703/) (2022)
- [InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning](https://mlsanity.com/paper/2205.12673/) (2022)
- [Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning](https://mlsanity.com/paper/2211.03044/) (2022)
- [Dynamic Prefix-Tuning for Generative Template-based Event Extraction](https://mlsanity.com/paper/2205.06166/) (2022)
- [How are Prompts Different in Terms of Sensitivity?](https://mlsanity.com/paper/2311.07230/) (2023)
- [Large Language Model Instruction Following: A Survey of Progresses and Challenges](https://mlsanity.com/paper/2303.10475/) (2023)
- [Training Models to Generate, Recognize, and Reframe Unhelpful Thoughts](https://mlsanity.com/paper/2307.02768/) (2023)
- [Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectives](https://mlsanity.com/paper/2307.16851/) (2023)
- [CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation](https://mlsanity.com/paper/2204.00862/) (2022)
- [Source2Synth: Synthetic Data Generation and Curation Grounded in Real Data Sources](https://mlsanity.com/paper/2409.08239/) (2024)
- [Textual Entailment for Event Argument Extraction: Zero- and Few-Shot with Multi-Source Learning](https://mlsanity.com/paper/2205.01376/) (2022)
- [An Overview on Language Models: Recent Developments and Outlook](https://mlsanity.com/paper/2303.05759/) (2023)
- [Dialogue Summaries as Dialogue States (DS2), Template-Guided Summarization for Few-shot Dialogue State Tracking](https://mlsanity.com/paper/2203.01552/) (2022)
- [Exploring Prompt-based Few-shot Learning for Grounded Dialog Generation](https://mlsanity.com/paper/2109.06513/) (2021)
- [MEAL: Stable and Active Learning for Few-Shot Prompting](https://mlsanity.com/paper/2211.08358/) (2022)

## Abstract (arXiv)

Providing pretrained language models with simple task descriptions in natural language enables them to solve some tasks in a fully unsupervised fashion. Moreover, when combined with regular learning from examples, this idea yields impressive few-shot results for a wide range of text classification tasks. It is also a promising direction to improve data efficiency in generative settings, but there are several challenges to using a combination of task descriptions and example-based learning for text generation. In particular, it is crucial to find task descriptions that are easy to understand for the pretrained model and to ensure that it actually makes good use of them; furthermore, effective measures against overfitting have to be implemented. In this paper, we show how these challenges can be tackled: We introduce GenPET, a method for text generation that is based on pattern-exploiting training, a recent approach for combining textual instructions with supervised learning that only works for classification tasks. On several summarization and headline generation datasets, GenPET gives consistent improvements over strong baselines in few-shot settings.