mlsanityData to 7 Oct 2026

Text-to-Text Pre-Training for Data-to-Text Tasks

arXiv 2005.10433 ↗First posted 21 May 2020cs.CLcited by 104

Not on the map. Why?

Abstract (arXiv)

We study the pre-train + fine-tune strategy for data-to-text tasks. Our experiments indicate that text-to-text pre-training in the form of T5, enables simple, end-to-end transformer based models to outperform pipelined neural architectures tailored for data-to-text generation, as well as alternative language model based pre-training techniques such as BERT and GPT-2. Importantly, T5 pre-training leads to better generalization, as evidenced by large improvements on out-of-domain test sets. We hope our work serves as a useful baseline for future research, as transfer learning becomes ever more prevalent for data-to-text tasks.

Key facts. Cited by 104 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 29; 2022: 29; 2023: 25; 2024: 16; 2025: 5.

104 papers.

Sort:

By area:

  1. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  2. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  3. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  4. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  5. 2021FeTaQA: Free-form Table Question Answeringcited by 174
  6. 2022A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Modelscited by 128
  7. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  8. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  9. 2023AUDIT: Audio Editing by Following Instructions with Latent Diffusion Modelscited by 81
  10. 2022Understanding Transformer Memorization Recall Through Idiomscited by 62
  11. 2021JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphscited by 58
  12. 2023One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Eracited by 55
38 more
  1. 2021Plan-then-Generate: Controlled Data-to-Text Generation via Planningcited by 52
  2. 2021Structural Adapters in Pretrained Language Models for AMR-to-text Generationcited by 48
  3. 2022Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Taskscited by 42
  4. 2022IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languagescited by 42
  5. 2024MixDQ: Memory-Efficient Few-Step Text-to-Image Diffusion Models with Metric-Decoupled Mixed Precision Quantizationcited by 42
  6. 2021Controlling Hallucinations at Word Level in Data-to-Text Generationcited by 41
  7. 2021IndicBART: A Pre-trained Model for Indic Natural Language Generationcited by 39
  8. 2022GEMv2: Multilingual NLG Benchmarking in a Single Line of Codecited by 36
  9. 2024Large Language Models for UAVs: Current State and Pathways to the Futurecited by 33
  10. 2022Robust (Controlled) Table-to-Text Generation with Structure-Aware Equivariance Learningcited by 32
  11. 2021CINS: Comprehensive Instruction for Few-shot Learning in Task-oriented Dialog Systemscited by 30
  12. 2021On Training Instance Selection for Few-Shot Neural Text Generationcited by 25
  13. 2023Using Large Language Models for Zero-Shot Natural Language Generation from Knowledge Graphscited by 24
  14. 2021Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluationcited by 21
  15. 2021Few-Shot Table-to-Text Generation with Prototype Memorycited by 21
  16. 2022CoNT: Contrastive Neural Text Generationcited by 21
  17. 2022Table-To-Text generation and pre-training with TabT5cited by 20
  18. 2021Improving Compositional Generalization with Self-Training for Data-to-Text Generationcited by 19
  19. 2022Neural Pipeline for Zero-Shot Data-to-Text Generationcited by 19
  20. 2021Automatic Construction of Evaluation Suites for Natural Language Generation Datasetscited by 18
  21. 2021SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasetscited by 18
  22. 2022GAP: A Graph-aware Language Model Framework for Knowledge Graph-to-Text Generationcited by 16
  23. 2022PLOG: Table-to-Logic Pretraining for Logical Table-to-Text Generationcited by 16
  24. 2022ASDOT: Any-Shot Data-to-Text Generation with Pretrained Language Modelscited by 14
  25. 2024Unifying Structured Data as Graph for Data-to-Text Pre-Trainingcited by 13
  26. 2025Object-AVEdit: An Object-level Audio-Visual Editing Modelcited by 12
  27. 2021Stage-wise Fine-tuning for Graph-to-Text Generationcited by 11
  28. 2021DYPLOC: Dynamic Planning of Content Using Mixed Language Models for Text Generationcited by 11
  29. 2021Attention Is Indeed All You Need: Semantically Attention-Guided Decoding for Data-to-Text NLGcited by 11
  30. 2021GenNI: Human-AI Collaboration for Data-Backed Text Generationcited by 11
  31. 2023The Iron(ic) Melting Pot: Reviewing Human Evaluation in Humour, Irony and Sarcasm Generationcited by 11
  32. 2021Biomedical Data-to-Text Generation via Fine-Tuning Transformerscited by 9
  33. 2022Learning a Better Initialization for Soft Prompts via Meta-Learningcited by 9
  34. 2023TextDefense: Adversarial Text Detection based on Word Importance Entropycited by 9
  35. 2022What Makes Data-to-Text Generation Hard for Pretrained Language Models?cited by 8
  36. 2022R2D2: Robust Data-to-Text with Replacement Detectioncited by 8
  37. 2022AARGH! End-to-end Retrieval-Generation for Task-Oriented Dialogcited by 8
  38. 2022Self-supervised Graph Masking Pre-training for Graph-to-Text Generationcited by 8

50 most cited of 104 All 104 (JSON)