# On Training Instance Selection for Few-Shot Neural Text Generation

arXiv: https://arxiv.org/abs/2107.03176 · HTML: https://mlsanity.com/paper/2107.03176/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 7 Jul 2021.

## Builds on (most cited first, 21 of 21)

- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [The E2E Dataset: New Challenges For End-to-End Generation](https://mlsanity.com/paper/1706.09254/) (2017)
- [Learning to Ask: Neural Question Generation for Reading Comprehension](https://mlsanity.com/paper/1705.00106/) (2017)
- [Text-to-Text Pre-Training for Data-to-Text Tasks](https://mlsanity.com/paper/2005.10433/) (2020)
- [Learning Active Learning from Data](https://mlsanity.com/paper/1703.03365/) (2017)
- [Few-Shot NLG with Pre-Trained Language Model](https://mlsanity.com/paper/1904.09521/) (2019)
- [Semantic Redundancies in Image-Classification Datasets: The 10% You Don't Need](https://mlsanity.com/paper/1901.11409/) (2019)
- [Does the Order of Training Samples Matter? Improving Neural Data-to-Text Generation with Curriculum Learning](https://mlsanity.com/paper/2102.03554/) (2021)
- [Explaining and Improving Model Behavior with k Nearest Neighbor Representations](https://mlsanity.com/paper/2010.09030/) (2020)
- [Neural Data-to-Text Generation with LM-based Text Augmentation](https://mlsanity.com/paper/2102.03556/) (2021)
- [DART: A Lightweight Quality-Suggestive Data-to-Text Annotation Tool](https://mlsanity.com/paper/2010.04141/) (2020)
- [Jointly Improving Language Understanding and Generation with Quality-Weighted Weak Supervision of Automatic Labeling](https://mlsanity.com/paper/2102.03551/) (2021)
- [Learning From Less Data: Diversified Subset Selection and Active Learning in Image Classification Tasks](https://mlsanity.com/paper/1805.11191/) (2018)
- [Enhancing Sentence Embedding with Generalized Pooling](https://mlsanity.com/paper/1806.09828/) (2018)

## Built on by (most cited first, 26 of 26)

- [A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challenges](https://mlsanity.com/paper/2204.04859/) (2022)
- [Generating Representative Samples for Few-Shot Classification](https://mlsanity.com/paper/2205.02918/) (2022)
- [ConstitutionMaker: Interactively Critiquing Large Language Models by Converting Feedback into Principles](https://mlsanity.com/paper/2310.15428/) (2023)
- [ScatterShot: Interactive In-context Example Curation for Text Transformation](https://mlsanity.com/paper/2302.07346/) (2023)
- [Neural Pipeline for Zero-Shot Data-to-Text Generation](https://mlsanity.com/paper/2203.16279/) (2022)
- [Cold-Start Data Selection for Few-shot Language Model Fine-tuning: A Prompt-Based Uncertainty Propagation Approach](https://mlsanity.com/paper/2209.06995/) (2022)
- [A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomness](https://mlsanity.com/paper/2312.01082/) (2023)
- [Towards Free Data Selection with General-Purpose Models](https://mlsanity.com/paper/2309.17342/) (2023)
- [Active Few-Shot Learning for Text Classification](https://mlsanity.com/paper/2502.18782/) (2025)
- [Attend, Memorize and Generate: Towards Faithful Table-to-Text Generation in Few Shots](https://mlsanity.com/paper/2203.00732/) (2022)
- [Decorate the Examples: A Simple Method of Prompt Design for Biomedical Relation Extraction](https://mlsanity.com/paper/2204.10360/) (2022)
- [A Systematic Review of Data-to-Text NLG](https://mlsanity.com/paper/2402.08496/) (2024)
- [Active Learning for Natural Language Generation](https://mlsanity.com/paper/2305.15040/) (2023)
- [In-Context Learning on a Budget: A Case Study in Token Classification](https://mlsanity.com/paper/2406.13274/) (2024)
- [Leveraging Encoder-only Large Language Models for Mobile App Review Feature Extraction](https://mlsanity.com/paper/2408.01063/) (2024)
- [DEUCE: Dual-diversity Enhancement and Uncertainty-awareness for Cold-start Active Learning](https://mlsanity.com/paper/2502.00305/) (2025)
- [YaleNLP @ PerAnsSumm 2025: Multi-Perspective Integration via Mixture-of-Agents for Enhanced Healthcare QA Summarization](https://mlsanity.com/paper/2504.03932/) (2025)
- [PromptAL: Sample-Aware Dynamic Soft Prompts for Few-Shot Active Learning](https://mlsanity.com/paper/2507.16424/) (2025)
- [Self-training with Two-phase Self-augmentation for Few-shot Dialogue Generation](https://mlsanity.com/paper/2205.09661/) (2022)
- [What Makes Good Few-shot Examples for Vision-Language Models?](https://mlsanity.com/paper/2405.13532/) (2024)
- [Few-Shot Optimization for Sensor Data Using Large Language Models: A Case Study on Fatigue Detection](https://mlsanity.com/paper/2505.18754/) (2025)
- [Reinforcement Learning for Few-Shot Text Generation Adaptation](https://mlsanity.com/paper/2111.11030/) (2021)
- [Revisiting Sample Size Determination in Natural Language Understanding](https://mlsanity.com/paper/2307.00374/) (2023)
- [AST-MHSA : Code Summarization using Multi-Head Self-Attention](https://mlsanity.com/paper/2308.05646/) (2023)
- [Modeling Data Diversity for Joint Instance and Verbalizer Selection in Cold-Start Scenarios](https://mlsanity.com/paper/2507.00330/) (2025)
- [Verb-ICL: Rethinking In-Context Learning for Structured Prediction](https://mlsanity.com/paper/2610.04725/) (2026)

## Abstract (arXiv)

Large-scale pretrained language models have led to dramatic improvements in text generation. Impressive performance can be achieved by finetuning only on a small number of instances (few-shot setting). Nonetheless, almost all previous work simply applies random sampling to select the few-shot training instances. Little to no attention has been paid to the selection strategies and how they would affect model performance. In this work, we present a study on training instance selection in few-shot neural text generation. The selection decision is made based only on the unlabeled data so as to identify the most worthwhile data points that should be annotated under some budget of labeling cost. Based on the intuition that the few-shot training instances should be diverse and representative of the entire data distribution, we propose a simple selection strategy with K-means clustering. We show that even with the naive clustering-based approach, the generation models consistently outperform random sampling on three text generation tasks: data-to-text generation, document summarization and question generation. We hope that this work will call for more attention on this largely unexplored area.