# Cross-Domain, Multi-Task Data-to-Text Generation without In-Domain Training Data

arXiv: https://arxiv.org/abs/2608.23391 · HTML: https://mlsanity.com/paper/2608.23391/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 75 references that are in the map.
- First posted on arXiv 24 Aug 2026.

## Builds on (most cited first, 30 of 33)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning](https://mlsanity.com/paper/2408.05517/) (2024)
- [ToTTo: A Controlled Table-To-Text Generation Dataset](https://mlsanity.com/paper/2004.14373/) (2020)
- [The E2E Dataset: New Challenges For End-to-End Generation](https://mlsanity.com/paper/1706.09254/) (2017)
- [Challenges in Data-to-Document Generation](https://mlsanity.com/paper/1707.08052/) (2017)
- [Knowledge Distillation Meets Self-Supervision](https://mlsanity.com/paper/2006.07114/) (2020)
- [Data-to-Text Generation with Content Selection and Planning](https://mlsanity.com/paper/1809.00582/) (2018)
- [Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Model](https://mlsanity.com/paper/2010.09142/) (2020)
- [Neural AMR: Sequence-to-Sequence Models for Parsing and Generation](https://mlsanity.com/paper/1704.08381/) (2017)
- [Table-to-text Generation by Structure-aware Seq2seq Learning](https://mlsanity.com/paper/1711.09724/) (2017)
- [Dialogue Chain-of-Thought Distillation for Commonsense-aware Conversational Agents](https://mlsanity.com/paper/2310.09343/) (2023)
- [Time Weaver: A Conditional Time Series Generation Model](https://mlsanity.com/paper/2403.02682/) (2024)
- [Noisy Self-Knowledge Distillation for Text Summarization](https://mlsanity.com/paper/2009.07032/) (2020)
- [Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasing](https://mlsanity.com/paper/2305.16635/) (2023)
- [MixSKD: Self-Knowledge Distillation from Mixup for Image Recognition](https://mlsanity.com/paper/2208.05768/) (2022)
- [Findings of the E2E NLG Challenge](https://mlsanity.com/paper/1810.01170/) (2018)
- [Effective Distillation of Table-based Reasoning Ability from LLMs](https://mlsanity.com/paper/2309.13182/) (2023)
- [A Hierarchical Model for Data-to-Text Generation](https://mlsanity.com/paper/1912.10011/) (2019)
- [A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training](https://mlsanity.com/paper/2305.02031/) (2023)
- [Beyond Traditional Benchmarks: Analyzing Behaviors of Open LLMs on Data-to-Text Generation](https://mlsanity.com/paper/2401.10186/) (2024)
- [ASDOT: Any-Shot Data-to-Text Generation with Pretrained Language Models](https://mlsanity.com/paper/2210.04325/) (2022)
- [Data-to-Text Generation with Iterative Text Editing](https://mlsanity.com/paper/2011.01694/) (2020)

## Abstract (arXiv)

Structured data exists in many forms (tables, knowledge graphs, charts, and time series), and converting it into text may involve different generation tasks. However, most prior work on data-to-text (D2T) generation has focused on specific tasks and datasets, relying either on task-specific training data or on the zero-shot capabilities of large language models. We study cross-domain D2T generation in a setting where neither in-domain training text nor test references are available, and where domains, generation goals, and input structures vary substantially. We compare data-driven knowledge distillation (DDKD) against zero-shot inference and fine-tuning on out-of-domain D2T data, and introduce structure-preserving augmentation via structural subsampling and perturbation. Experiments on five benchmarks show that, at constant model size (1.7B parameters), DDKD consistently outperforms both fine-tuning and zero-shot inference. Moreover, the resulting small models outperform a much larger finetuned model on two of the five domains, achieving comparable performance on the remaining three. We further construct QUINTD-5, a fivefold extension of QUINTD-1, and show that simply scaling real target-domain inputs yields only modest gains, whereas our augmentation strategy remains more effective and more cost-efficient for cross-domain distillation.