# Improving Compositional Generalization with Self-Training for Data-to-Text Generation

arXiv: https://arxiv.org/abs/2110.08467 · HTML: https://mlsanity.com/paper/2110.08467/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 45 references that are in the map.
- First posted on arXiv 16 Oct 2021.

## Builds on (most cited first, 30 of 33)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset](https://mlsanity.com/paper/1909.05855/) (2019)
- [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019)
- [COGS: A Compositional Generalization Challenge Based on Semantic Interpretation](https://mlsanity.com/paper/2010.05465/) (2020)
- [Good-Enough Compositional Data Augmentation](https://mlsanity.com/paper/1904.09545/) (2019)
- [Revisiting Self-Training for Neural Sequence Generation](https://mlsanity.com/paper/1909.13788/) (2019)
- [Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation](https://mlsanity.com/paper/1911.03842/) (2019)
- [Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems](https://mlsanity.com/paper/1508.01745/) (2015)
- [Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers](https://mlsanity.com/paper/2109.10686/) (2021)
- [Text-to-Text Pre-Training for Data-to-Text Tasks](https://mlsanity.com/paper/2005.10433/) (2020)
- [Compositional generalization through meta sequence-to-sequence learning](https://mlsanity.com/paper/1906.05381/) (2019)
- [Few-shot Natural Language Generation for Task-Oriented Dialog](https://mlsanity.com/paper/2002.12328/) (2020)
- [Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge](https://mlsanity.com/paper/1901.07931/) (2019)
- [Measuring Compositionality in Representation Learning](https://mlsanity.com/paper/1902.07181/) (2019)
- [Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation](https://mlsanity.com/paper/1904.03396/) (2019)
- [Learning to Recombine and Resample Data for Compositional Generalization](https://mlsanity.com/paper/2010.03706/) (2020)
- [Felix: Flexible Text Editing Through Tagging and Insertion](https://mlsanity.com/paper/2003.10687/) (2020)
- [Neural AMR: Sequence-to-Sequence Models for Parsing and Generation](https://mlsanity.com/paper/1704.08381/) (2017)
- [Neural data-to-text generation: A comparison between pipeline and end-to-end architectures](https://mlsanity.com/paper/1908.09022/) (2019)
- [Constrained Decoding for Neural NLG from Compositional Representations in Task-Oriented Dialogue](https://mlsanity.com/paper/1906.07220/) (2019)
- [Findings of the E2E NLG Challenge](https://mlsanity.com/paper/1810.01170/) (2018)
- [Deep Graph Convolutional Encoders for Structured Data to Text Generation](https://mlsanity.com/paper/1810.09995/) (2018)
- [Neural Data-to-Text Generation via Jointly Learning the Segmentation and Correspondence](https://mlsanity.com/paper/2005.01096/) (2020)

## Built on by (most cited first, 19 of 19)

- [Benchmarking Complex Instruction-Following with Multiple Constraints Composition](https://mlsanity.com/paper/2407.03978/) (2024)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [DSI++: Updating Transformer Memory with New Documents](https://mlsanity.com/paper/2212.09744/) (2022)
- [Neural Pipeline for Zero-Shot Data-to-Text Generation](https://mlsanity.com/paper/2203.16279/) (2022)
- [ASDOT: Any-Shot Data-to-Text Generation with Pretrained Language Models](https://mlsanity.com/paper/2210.04325/) (2022)
- [Unifying Structured Data as Graph for Data-to-Text Pre-Training](https://mlsanity.com/paper/2401.01183/) (2024)
- [Deciphering the Role of Representation Disentanglement: Investigating Compositional Generalization in CLIP Models](https://mlsanity.com/paper/2407.05897/) (2024)
- [TaTa: A Multilingual Table-to-Text Dataset for African Languages](https://mlsanity.com/paper/2211.00142/) (2022)
- [Seen to Unseen: Exploring Compositional Generalization of Multi-Attribute Controllable Dialogue Generation](https://mlsanity.com/paper/2306.10317/) (2023)
- [Few Shot Rationale Generation using Self-Training with Dual Teachers](https://mlsanity.com/paper/2306.03315/) (2023)
- [SPOR: A Comprehensive and Practical Evaluation Method for Compositional Generalization in Data-to-Text Generation](https://mlsanity.com/paper/2405.10650/) (2024)
- [Ontology-Free General-Domain Knowledge Graph-to-Text Generation Dataset Synthesis using Large Language Model](https://mlsanity.com/paper/2409.07088/) (2024)
- [Data-efficient Active Learning for Structured Prediction with Partial Annotation and Self-Training](https://mlsanity.com/paper/2305.12634/) (2023)
- [You Can Generate It Again: Data-to-Text Generation with Verification and Correction Prompting](https://mlsanity.com/paper/2306.15933/) (2023)
- [Self-training with Two-phase Self-augmentation for Few-shot Dialogue Generation](https://mlsanity.com/paper/2205.09661/) (2022)
- [Joint Dropout: Improving Generalizability in Low-Resource Neural Machine Translation through Phrase Pair Variables](https://mlsanity.com/paper/2307.12835/) (2023)
- [Compositional Generalization for Data-to-Text Generation](https://mlsanity.com/paper/2312.02748/) (2023)
- [Self-training from Self-memory in Data-to-text Generation](https://mlsanity.com/paper/2401.10567/) (2024)
- [State Value Generation with Prompt Learning and Self-Training for Low-Resource Dialogue State Tracking](https://mlsanity.com/paper/2401.16862/) (2024)

## Abstract (arXiv)

Data-to-text generation focuses on generating fluent natural language responses from structured meaning representations (MRs). Such representations are compositional and it is costly to collect responses for all possible combinations of atomic meaning schemata, thereby necessitating few-shot generalization to novel MRs. In this work, we systematically study the compositional generalization of the state-of-the-art T5 models in few-shot data-to-text tasks. We show that T5 models fail to generalize to unseen MRs, and we propose a template-based input representation that considerably improves the model's generalization capability. To further improve the model's performance, we propose an approach based on self-training using fine-tuned BLEURT for pseudo response selection. On the commonly-used SGD and Weather benchmarks, the proposed self-training approach improves tree accuracy by 46%+ and reduces the slot error rates by 73%+ over the strong T5 baselines in few-shot settings.