# Curriculum-Based Self-Training Makes Better Few-Shot Learners for Data-to-Text Generation

arXiv: https://arxiv.org/abs/2206.02712 · HTML: https://mlsanity.com/paper/2206.02712/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 34 references that are in the map.
- First posted on arXiv 6 Jun 2022.

## Builds on (most cited first, 13 of 13)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Certified Robustness to Adversarial Word Substitutions](https://mlsanity.com/paper/1909.00986/) (2019)
- [Revisiting Self-Training for Neural Sequence Generation](https://mlsanity.com/paper/1909.13788/) (2019)
- [Investigating Pretrained Language Models for Graph-to-Text Generation](https://mlsanity.com/paper/2007.08426/) (2020)
- [Text-to-Text Pre-Training for Data-to-Text Tasks](https://mlsanity.com/paper/2005.10433/) (2020)
- [Self-training Improves Pre-training for Natural Language Understanding](https://mlsanity.com/paper/2010.02194/) (2020)
- [JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs](https://mlsanity.com/paper/2106.10502/) (2021)
- [Few-Shot NLG with Pre-Trained Language Model](https://mlsanity.com/paper/1904.09521/) (2019)
- [Modeling Global and Local Node Contexts for Text Generation from Knowledge Graphs](https://mlsanity.com/paper/2001.11003/) (2020)
- [End-to-End Content and Plan Selection for Data-to-Text Generation](https://mlsanity.com/paper/1810.04700/) (2018)
- [Promoting Graph Awareness in Linearized Graph-to-Text Generation](https://mlsanity.com/paper/2012.15793/) (2020)

## Built on by (most cited first, 2 of 2)

- [Compositional Generalization for Data-to-Text Generation](https://mlsanity.com/paper/2312.02748/) (2023)
- [Self-training from Self-memory in Data-to-text Generation](https://mlsanity.com/paper/2401.10567/) (2024)

## Abstract (arXiv)

Despite the success of text-to-text pre-trained models in various natural language generation (NLG) tasks, the generation performance is largely restricted by the number of labeled data in downstream tasks, particularly in data-to-text generation tasks. Existing works mostly utilize abundant unlabeled structured data to conduct unsupervised pre-training for task adaption, which fail to model the complex relationship between source structured data and target texts. Thus, we introduce self-training as a better few-shot learner than task-adaptive pre-training, which explicitly captures this relationship via pseudo-labeled data generated by the pre-trained model. To alleviate the side-effect of low-quality pseudo-labeled data during self-training, we propose a novel method called Curriculum-Based Self-Training (CBST) to effectively leverage unlabeled data in a rearranged order determined by the difficulty of text generation. Experimental results show that our method can outperform fine-tuning and task-adaptive pre-training methods, and achieve state-of-the-art performance in the few-shot setting of data-to-text generation.