# PCC: Paraphrasing with Bottom-k Sampling and Cyclic Learning for Curriculum Data Augmentation

arXiv: https://arxiv.org/abs/2208.08110 · HTML: https://mlsanity.com/paper/2208.08110/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 64 references that are in the map.
- First posted on arXiv 17 Aug 2022.

## Builds on (most cited first, 30 of 41)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://mlsanity.com/paper/1610.02424/) (2016)
- [Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization](https://mlsanity.com/paper/1807.00230/) (2018)
- [Competence-based Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/1903.09848/) (2019)
- [TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents](https://mlsanity.com/paper/1901.08149/) (2019)
- [ParlAI: A Dialog Research Software Platform](https://mlsanity.com/paper/1705.06476/) (2017)
- [Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks](https://mlsanity.com/paper/1802.03796/) (2018)
- [Conditional BERT Contextual Augmentation](https://mlsanity.com/paper/1812.06705/) (2018)
- [A Dataset for Document Grounded Conversations](https://mlsanity.com/paper/1809.07358/) (2018)
- [ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons](https://mlsanity.com/paper/1909.03087/) (2019)
- [Edinburgh Neural Machine Translation Systems for WMT 16](https://mlsanity.com/paper/1606.02891/) (2016)
- [SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation](https://mlsanity.com/paper/1808.07512/) (2018)
- [Exploiting Persona Information for Diverse Generation of Conversational Responses](https://mlsanity.com/paper/1905.12188/) (2019)
- [Skeleton-to-Response: Dialogue Generation Guided by Retrieval Memory](https://mlsanity.com/paper/1809.05296/) (2018)
- [Paraphrase Augmented Task-Oriented Dialog Generation](https://mlsanity.com/paper/2004.07462/) (2020)
- [On the Importance of Word and Sentence Representation Learning in Implicit Discourse Relation Classification](https://mlsanity.com/paper/2004.12617/) (2020)
- [Few-Shot Text Classification with Triplet Networks, Data Augmentation, and Curriculum Learning](https://mlsanity.com/paper/2103.07552/) (2021)
- [Data Manipulation: Towards Effective Instance Learning for Neural Dialogue Generation via Learning to Augment and Reweight](https://mlsanity.com/paper/2004.02594/) (2020)

## Built on by (most cited first, 14 of 14)

- [Curricular Object Manipulation in LiDAR-based Object Detection](https://mlsanity.com/paper/2304.04248/) (2023)
- [Are Robust LLM Fingerprints Adversarially Robust?](https://mlsanity.com/paper/2509.26598/) (2025)
- [Diffusion Curriculum: Synthetic-to-Real Data Curriculum via Image-Guided Diffusion](https://mlsanity.com/paper/2410.13674/) (2024)
- [A Survey of Event Causality Identification: Taxonomy, Challenges, Assessment, and Prospects](https://mlsanity.com/paper/2411.10371/) (2024)
- [Dictionary Insertion Prompting for Multilingual Reasoning on Multilingual Large Language Models](https://mlsanity.com/paper/2411.01141/) (2024)
- [EPA: Easy Prompt Augmentation on Large Language Models via Multiple Sources and Multiple Targets](https://mlsanity.com/paper/2309.04725/) (2023)
- [Colorful Cutout: Enhancing Image Data Augmentation with Curriculum Learning](https://mlsanity.com/paper/2403.20012/) (2024)
- [Enhancing Effectiveness and Robustness in a Low-Resource Regime via Decision-Boundary-aware Data Augmentation](https://mlsanity.com/paper/2403.15512/) (2024)
- [Task-Oriented Paraphrase Analytics](https://mlsanity.com/paper/2403.17564/) (2024)
- [Toxic Subword Pruning for Dialogue Response Generation on Large Language Models](https://mlsanity.com/paper/2410.04155/) (2024)
- [Adverb Is the Key: Simple Text Data Augmentation with Adverb Deletion](https://mlsanity.com/paper/2403.20015/) (2024)
- [SLoW: Select Low-frequency Words! Automatic Dictionary Selection for Translation on Large Language Models](https://mlsanity.com/paper/2507.18902/) (2025)
- [Adam's Law: Textual Frequency Law on Large Language Models](https://mlsanity.com/paper/2604.02176/) (2026)
- [Disentangling Curriculum Learning in NLP: Towards a Unifying Taxonomy](https://mlsanity.com/paper/2607.18984/) (2026)

## Abstract (arXiv)

Curriculum Data Augmentation (CDA) improves neural models by presenting synthetic data with increasing difficulties from easy to hard. However, traditional CDA simply treats the ratio of word perturbation as the difficulty measure and goes through the curriculums only once. This paper presents PCC: Paraphrasing with Bottom-k Sampling and Cyclic Learning for Curriculum Data Augmentation, a novel CDA framework via paraphrasing, which exploits the textual paraphrase similarity as the curriculum difficulty measure. We propose a curriculum-aware paraphrase generation module composed of three units: a paraphrase candidate generator with bottom-k sampling, a filtering mechanism and a difficulty measure. We also propose a cyclic learning strategy that passes through the curriculums multiple times. The bottom-k sampling is proposed to generate super-hard instances for the later curriculums. Experimental results on few-shot text classification as well as dialogue generation indicate that PCC surpasses competitive baselines. Human evaluation and extensive case studies indicate that bottom-k sampling effectively generates super-hard instances, and PCC significantly improves the baseline dialogue agent.