# Neural Data-to-Text Generation with LM-based Text Augmentation

arXiv: https://arxiv.org/abs/2102.03556 · HTML: https://mlsanity.com/paper/2102.03556/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 56 references that are in the map.
- First posted on arXiv 6 Feb 2021.

## Builds on (most cited first, 30 of 34)

- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://mlsanity.com/paper/1711.00043/) (2017)
- [The E2E Dataset: New Challenges For End-to-End Generation](https://mlsanity.com/paper/1706.09254/) (2017)
- [Challenges in Data-to-Document Generation](https://mlsanity.com/paper/1707.08052/) (2017)
- [Unsupervised Neural Machine Translation](https://mlsanity.com/paper/1710.11041/) (2017)
- [Dual Learning for Machine Translation](https://mlsanity.com/paper/1611.00179/) (2016)
- [Phrase-Based & Neural Unsupervised Machine Translation](https://mlsanity.com/paper/1804.07755/) (2018)
- [Style Transfer Through Back-Translation](https://mlsanity.com/paper/1804.09000/) (2018)
- [Hello, It's GPT-2 -- How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1907.05774/) (2019)
- [Few-shot Natural Language Generation for Task-Oriented Dialog](https://mlsanity.com/paper/2002.12328/) (2020)
- [Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings](https://mlsanity.com/paper/1811.01136/) (2018)
- [A Probabilistic Formulation of Unsupervised Text Style Transfer](https://mlsanity.com/paper/2002.03912/) (2020)
- [Few-Shot NLG with Pre-Trained Language Model](https://mlsanity.com/paper/1904.09521/) (2019)
- [DARE: Data Augmented Relation Extraction with GPT-2](https://mlsanity.com/paper/2004.13845/) (2020)
- [SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression](https://mlsanity.com/paper/1904.03651/) (2019)
- [Neural Data-to-Text Generation via Jointly Learning the Segmentation and Correspondence](https://mlsanity.com/paper/2005.01096/) (2020)
- [A Multilingual View of Unsupervised Machine Translation](https://mlsanity.com/paper/2002.02955/) (2020)
- [A Deep Ensemble Model with Slot Alignment for Sequence-to-Sequence Natural Language Generation](https://mlsanity.com/paper/1805.06553/) (2018)
- [Multiple-Attribute Text Style Transfer](https://mlsanity.com/paper/1811.00552/) (2018)
- [Key Fact as Pivot: A Two-Stage Model for Low Resource Table-to-Text Generation](https://mlsanity.com/paper/1908.03067/) (2019)
- [DART: A Lightweight Quality-Suggestive Data-to-Text Annotation Tool](https://mlsanity.com/paper/2010.04141/) (2020)
- [Unsupervised Natural Language Generation with Denoising Autoencoders](https://mlsanity.com/paper/1804.07899/) (2018)
- [Semi-Supervised Neural Text Generation by Joint Learning of Natural Language Generation and Natural Language Understanding Models](https://mlsanity.com/paper/1910.03484/) (2019)
- [Unsupervised Pidgin Text Generation By Pivoting English Data and Self-Training](https://mlsanity.com/paper/2003.08272/) (2020)
- [Towards Unsupervised Language Understanding and Generation by Joint Dual Learning](https://mlsanity.com/paper/2004.14710/) (2020)
- [Char2char Generation with Reranking for the E2E NLG Challenge](https://mlsanity.com/paper/1811.05826/) (2018)

## Built on by (most cited first, 22 of 22)

- [A Survey on Data Augmentation for Text Classification](https://mlsanity.com/paper/2107.03158/) (2021)
- [A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challenges](https://mlsanity.com/paper/2204.04859/) (2022)
- [On Training Instance Selection for Few-Shot Neural Text Generation](https://mlsanity.com/paper/2107.03176/) (2021)
- [Neural Pipeline for Zero-Shot Data-to-Text Generation](https://mlsanity.com/paper/2203.16279/) (2022)
- [SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets](https://mlsanity.com/paper/2111.06467/) (2021)
- [Learning Fine-grained Fact-Article Correspondence in Legal Cases](https://mlsanity.com/paper/2104.10726/) (2021)
- [Recent Advances in Neural Text Generation: A Task-Agnostic Survey](https://mlsanity.com/paper/2203.03047/) (2022)
- [Domain-Specific Text Generation for Machine Translation](https://mlsanity.com/paper/2208.05909/) (2022)
- [Time-Aware Ancient Chinese Text Translation and Inference](https://mlsanity.com/paper/2107.03179/) (2021)
- [Counterfactual Data Augmentation via Perspective Transition for Open-Domain Dialogues](https://mlsanity.com/paper/2210.16838/) (2022)
- [The SelectGen Challenge: Finding the Best Training Samples for Few-Shot Neural Text Generation](https://mlsanity.com/paper/2108.06614/) (2021)
- [A Systematic Review of Data-to-Text NLG](https://mlsanity.com/paper/2402.08496/) (2024)
- [Improving Logical-Level Natural Language Generation with Topic-Conditioned Data Augmentation and Logical Form Generation](https://mlsanity.com/paper/2112.06240/) (2021)
- [WeaSuL: Weakly Supervised Dialogue Policy Learning: Reward Estimation for Multi-turn Dialogue](https://mlsanity.com/paper/2108.01487/) (2021)
- [Knowledge-Grounded Conversational Data Augmentation with Generative Conversational Networks](https://mlsanity.com/paper/2207.11363/) (2022)
- [Few-Shot Data-to-Text Generation via Unified Representation and Multi-Source Learning](https://mlsanity.com/paper/2308.05317/) (2023)
- [Medical Scientific Table-to-Text Generation with Human-in-the-Loop under the Data Sparsity Constraint](https://mlsanity.com/paper/2205.12368/) (2022)
- [Neural Data-to-Text Generation Based on Small Datasets: Comparing the Added Value of Two Semi-Supervised Learning Approaches on Top of a Large Language Model](https://mlsanity.com/paper/2207.06839/) (2022)
- [Reweighting Strategy based on Synthetic Data Identification for Sentence Similarity](https://mlsanity.com/paper/2208.13376/) (2022)
- [Learning from Multiple Sources for Data-to-Text and Text-to-Data](https://mlsanity.com/paper/2302.11269/) (2023)
- [Entity Aware Syntax Tree Based Data Augmentation for Natural Language Understanding](https://mlsanity.com/paper/2209.02267/) (2022)
- [Evaluation Metrics for Text Data Augmentation in NLP](https://mlsanity.com/paper/2402.06766/) (2024)

## Abstract (arXiv)

For many new application domains for data-to-text generation, the main obstacle in training neural models consists of a lack of training data. While usually large numbers of instances are available on the data side, often only very few text samples are available. To address this problem, we here propose a novel few-shot approach for this setting. Our approach automatically augments the data available for training by (i) generating new text samples based on replacing specific values by alternative ones from the same category, (ii) generating new text samples based on GPT-2, and (iii) proposing an automatic method for pairing the new text samples with data samples. As the text augmentation can introduce noise to the training data, we use cycle consistency as an objective, in order to make sure that a given data sample can be correctly reconstructed after having been formulated as text (and that text samples can be reconstructed from data). On both the E2E and WebNLG benchmarks, we show that this weakly supervised training paradigm is able to outperform fully supervised seq2seq models with less than 10% annotations. By utilizing all annotated data, our model can boost the performance of a standard seq2seq model by over 5 BLEU points, establishing a new state-of-the-art on both datasets.