# What Makes Data-to-Text Generation Hard for Pretrained Language Models? arXiv: https://arxiv.org/abs/2205.11505 ยท HTML: https://mlsanity.com/paper/2205.11505/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 33 of its 53 references that are in the map. - First posted on arXiv 23 May 2022. ## Builds on (most cited first, 30 of 33) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019) - [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019) - [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021) - [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021) - [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020) - [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021) - [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019) - [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021) - [The E2E Dataset: New Challenges For End-to-End Generation](https://mlsanity.com/paper/1706.09254/) (2017) - [Challenges in Data-to-Document Generation](https://mlsanity.com/paper/1707.08052/) (2017) - [Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation](https://mlsanity.com/paper/1703.09902/) (2017) - [DART: Open-Domain Structured Data Record to Text Generation](https://mlsanity.com/paper/2007.02871/) (2020) - [On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?](https://mlsanity.com/paper/2204.07931/) (2022) - [Investigating Pretrained Language Models for Graph-to-Text Generation](https://mlsanity.com/paper/2007.08426/) (2020) - [FaithDial: A Faithful Benchmark for Information-Seeking Dialogue](https://mlsanity.com/paper/2204.10757/) (2022) - [Text-to-Text Pre-Training for Data-to-Text Tasks](https://mlsanity.com/paper/2005.10433/) (2020) - [Do Massively Pretrained Language Models Make Better Storytellers?](https://mlsanity.com/paper/1909.10705/) (2019) - [Semantic Noise Matters for Neural Natural Language Generation](https://mlsanity.com/paper/1911.03905/) (2019) - [A Simple, Fast Diverse Decoding Algorithm for Neural Generation](https://mlsanity.com/paper/1611.08562/) (2016) - [Few-Shot NLG with Pre-Trained Language Model](https://mlsanity.com/paper/1904.09521/) (2019) - [Using Local Knowledge Graph Construction to Scale Seq2Seq Models to Multi-Document Inputs](https://mlsanity.com/paper/1910.08435/) (2019) - [GPT-too: A language-model-first approach for AMR-to-text generation](https://mlsanity.com/paper/2005.09123/) (2020) - [Have Your Text and Use It Too! End-to-End Neural Data-to-Text Generation with Semantic Fidelity](https://mlsanity.com/paper/2004.06577/) (2020) - [Generating More Interesting Responses in Neural Conversation Models with Distributional Constraints](https://mlsanity.com/paper/1809.01215/) (2018) - [What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment](https://mlsanity.com/paper/1509.00838/) (2015) ## Built on by (most cited first, 8 of 8) - [f-Divergence Minimization for Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/2307.15190/) (2023) - [Watermarking Conditional Text Generation for AI Detection: Unveiling Challenges and a Semantic-Aware Watermark Remedy](https://mlsanity.com/paper/2307.13808/) (2023) - [Mind the Labels: Describing Relations in Knowledge Graphs With Pretrained Models](https://mlsanity.com/paper/2210.07373/) (2022) - [Ontology-Free General-Domain Knowledge Graph-to-Text Generation Dataset Synthesis using Large Language Model](https://mlsanity.com/paper/2409.07088/) (2024) - [Few-Shot Data-to-Text Generation via Unified Representation and Multi-Source Learning](https://mlsanity.com/paper/2308.05317/) (2023) - [ASPIRO: Any-shot Structured Parsing-error-Induced ReprOmpting for Consistent Data-to-Text Generation](https://mlsanity.com/paper/2310.17877/) (2023) - [ADLM -- stega: A Universal Adaptive Token Selection Algorithm for Improving Steganographic Text Quality via Information Entropy](https://mlsanity.com/paper/2410.20825/) (2024) - [Text Data Integration](https://mlsanity.com/paper/2603.27055/) (2026) ## Abstract (arXiv) Expressing natural language descriptions of structured facts or relations -- data-to-text generation (D2T) -- increases the accessibility of structured knowledge repositories. Previous work shows that pre-trained language models(PLMs) perform remarkably well on this task after fine-tuning on a significant amount of task-specific training data. On the other hand, while auto-regressive PLMs can generalize from a few task examples, their efficacy at D2T is largely unexplored. Furthermore, we have an incomplete understanding of the limits of PLMs on D2T. In this work, we conduct an empirical study of both fine-tuned and auto-regressive PLMs on the DART multi-domain D2T dataset. We consider their performance as a function of the amount of task-specific data and how these data are incorporated into the models: zero and few-shot learning, and fine-tuning of model weights. In addition, we probe the limits of PLMs by measuring performance on subsets of the evaluation data: novel predicates and abstractive test examples. To improve the performance on these subsets, we investigate two techniques: providing predicate descriptions in the context and re-ranking generated candidates by information reflected in the source. Finally, we conduct a human evaluation of model errors and show that D2T generation tasks would benefit from datasets with more careful manual curation.