# Guiding Generative Language Models for Data Augmentation in Few-Shot Text Classification

arXiv: https://arxiv.org/abs/2111.09064 · HTML: https://mlsanity.com/paper/2111.09064/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 61 references that are in the map.
- First posted on arXiv 17 Nov 2021.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [Not Enough Data? Deep Learning to the Rescue!](https://mlsanity.com/paper/1911.03118/) (2019)
- [Data Augmentation using Pre-trained Transformer Models](https://mlsanity.com/paper/2003.02245/) (2020)
- [Data Augmentation for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1705.00440/) (2017)
- [Conditional BERT Contextual Augmentation](https://mlsanity.com/paper/1812.06705/) (2018)
- [Deceiving Google's Perspective API Built for Detecting Toxic Comments](https://mlsanity.com/paper/1702.08138/) (2017)
- [Generative Data Augmentation for Commonsense Reasoning](https://mlsanity.com/paper/2004.11546/) (2020)
- [Automatic Text Summarization of COVID-19 Medical Research Articles using BERT and GPT-2](https://mlsanity.com/paper/2006.01997/) (2020)
- [Learning to Answer by Learning to Ask: Getting the Best of GPT-2 and BERT Worlds](https://mlsanity.com/paper/1911.02365/) (2019)
- [Fluent Response Generation for Conversational Question Answering](https://mlsanity.com/paper/2005.10464/) (2020)
- [On Data Augmentation for Extreme Multi-label Classification](https://mlsanity.com/paper/2009.10778/) (2020)
- [Semi-supervised Classification for Natural Language Processing](https://mlsanity.com/paper/1409.7612/) (2014)
- [Topic-Centric Unsupervised Multi-Document Summarization of Scientific and News Articles](https://mlsanity.com/paper/2011.08072/) (2020)
- [A Sentence-level Hierarchical BERT Model for Document Classification with Limited Labelled Data](https://mlsanity.com/paper/2106.06738/) (2021)
- [The Devil is in the Details: Evaluating Limitations of Transformer-based Methods for Granular Tasks](https://mlsanity.com/paper/2011.01196/) (2020)
- [Predicting Themes within Complex Unstructured Texts: A Case Study on Safeguarding Reports](https://mlsanity.com/paper/2010.14584/) (2020)

## Built on by (most cited first, 14 of 14)

- [Language Models for Text Classification: Is In-Context Learning Enough?](https://mlsanity.com/paper/2403.17661/) (2024)
- [A Comprehensive Survey on Data Augmentation](https://mlsanity.com/paper/2405.09591/) (2024)
- [Empowering Large Language Models for Textual Data Augmentation](https://mlsanity.com/paper/2404.17642/) (2024)
- [I Learn Better If You Speak My Language: Understanding the Superior Performance of Fine-Tuning Large Language Models with LLM-Generated Responses](https://mlsanity.com/paper/2402.11192/) (2024)
- [TTIDA: Controllable Generative Data Augmentation via Text-to-Text and Text-to-Image Models](https://mlsanity.com/paper/2304.08821/) (2023)
- [Evolving Knowledge Distillation with Large Language Models and Active Learning](https://mlsanity.com/paper/2403.06414/) (2024)
- [Improving Small Language Models on PubMedQA via Generative Data Augmentation](https://mlsanity.com/paper/2305.07804/) (2023)
- [A Unified Framework for Generative Data Augmentation: A Comprehensive Survey](https://mlsanity.com/paper/2310.00277/) (2023)
- [Targeted Augmentation for Low-Resource Event Extraction](https://mlsanity.com/paper/2405.08729/) (2024)
- [EDDA: A Encoder-Decoder Data Augmentation Framework for Zero-Shot Stance Detection](https://mlsanity.com/paper/2403.15715/) (2024)
- [BERT Goes Off-Topic: Investigating the Domain Transfer Challenge using Genre Classification](https://mlsanity.com/paper/2311.16083/) (2023)
- [Reweighting Strategy based on Synthetic Data Identification for Sentence Similarity](https://mlsanity.com/paper/2208.13376/) (2022)
- [Into the Unknown: Generating Geospatial Descriptions for New Environments](https://mlsanity.com/paper/2406.19967/) (2024)
- [GPTA: Generative Prompt Tuning Assistant for Synergistic Downstream Neural Network Enhancement with LLMs](https://mlsanity.com/paper/2404.00189/) (2024)

## Abstract (arXiv)

Data augmentation techniques are widely used for enhancing the performance of machine learning models by tackling class imbalance issues and data sparsity. State-of-the-art generative language models have been shown to provide significant gains across different NLP tasks. However, their applicability to data augmentation for text classification tasks in few-shot settings have not been fully explored, especially for specialised domains. In this paper, we leverage GPT-2 (Radford A et al, 2019) for generating artificial training instances in order to improve classification performance. Our aim is to analyse the impact the selection process of seed training examples have over the quality of GPT-generated samples and consequently the classifier performance. We perform experiments with several seed selection strategies that, among others, exploit class hierarchical structures and domain expert selection. Our results show that fine-tuning GPT-2 in a handful of label instances leads to consistent classification improvements and outperform competitive baselines. Finally, we show that guiding this process through domain expert selection can lead to further improvements, which opens up interesting research avenues for combining generative models and active learning.