# Stage-wise Fine-tuning for Graph-to-Text Generation

arXiv: https://arxiv.org/abs/2105.08021 · HTML: https://mlsanity.com/paper/2105.08021/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 17 May 2021.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Convolutional Networks on Graphs for Learning Molecular Fingerprints](https://mlsanity.com/paper/1509.09292/) (2015)
- [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020)
- [TabFact: A Large-scale Dataset for Table-based Fact Verification](https://mlsanity.com/paper/1909.02164/) (2019)
- [Challenges in Data-to-Document Generation](https://mlsanity.com/paper/1707.08052/) (2017)
- [DART: Open-Domain Structured Data Record to Text Generation](https://mlsanity.com/paper/2007.02871/) (2020)
- [Text Generation from Knowledge Graphs with Graph Transformers](https://mlsanity.com/paper/1904.02342/) (2019)
- [Handling Divergent Reference Texts when Evaluating Table-to-Text Generation](https://mlsanity.com/paper/1906.01081/) (2019)
- [Investigating Pretrained Language Models for Graph-to-Text Generation](https://mlsanity.com/paper/2007.08426/) (2020)
- [Data-to-Text Generation with Content Selection and Planning](https://mlsanity.com/paper/1809.00582/) (2018)
- [Text-to-Text Pre-Training for Data-to-Text Tasks](https://mlsanity.com/paper/2005.10433/) (2020)
- [Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation](https://mlsanity.com/paper/1904.03396/) (2019)
- [A Graph-to-Sequence Model for AMR-to-Text Generation](https://mlsanity.com/paper/1805.02473/) (2018)
- [Neural data-to-text generation: A comparison between pipeline and end-to-end architectures](https://mlsanity.com/paper/1908.09022/) (2019)
- [Towards Faithful Neural Table-to-Text Generation with Content-Matching Constraints](https://mlsanity.com/paper/2005.00969/) (2020)
- [Have Your Text and Use It Too! End-to-End Neural Data-to-Text Generation with Semantic Fidelity](https://mlsanity.com/paper/2004.06577/) (2020)
- [Modeling Global and Local Node Contexts for Text Generation from Knowledge Graphs](https://mlsanity.com/paper/2001.11003/) (2020)
- [Deep Graph Convolutional Encoders for Structured Data to Text Generation](https://mlsanity.com/paper/1810.09995/) (2018)
- [Neural Data-to-Text Generation via Jointly Learning the Segmentation and Correspondence](https://mlsanity.com/paper/2005.01096/) (2020)
- [Enhancing AMR-to-Text Generation with Dual Graph Representations](https://mlsanity.com/paper/1909.00352/) (2019)
- [Describing a Knowledge Base](https://mlsanity.com/paper/1809.01797/) (2018)
- [Table-to-Text Generation with Effective Hierarchical Encoder on Three Dimensions (Row, Column and Time)](https://mlsanity.com/paper/1909.02304/) (2019)

## Built on by (most cited first, 11 of 11)

- [Self-supervised Graph Masking Pre-training for Graph-to-Text Generation](https://mlsanity.com/paper/2210.10599/) (2022)
- [The Effects of Hallucinations in Synthetic Training Data for Relation Extraction](https://mlsanity.com/paper/2410.08393/) (2024)
- [A Comparative Analysis of Conversational Large Language Models in Knowledge-Based Text Generation](https://mlsanity.com/paper/2402.01495/) (2024)
- [EEL: Efficiently Encoding Lattices for Reranking](https://mlsanity.com/paper/2306.00947/) (2023)
- [ADEPT: Adaptive Dynamic Early-Exit Process for Transformers](https://mlsanity.com/paper/2601.03700/) (2026)
- [Triples-to-Text Generation with Reinforcement Learning Based Graph-augmented Neural Networks](https://mlsanity.com/paper/2111.10545/) (2021)
- [XF2T: Cross-lingual Fact-to-Text Generation for Low-Resource Languages](https://mlsanity.com/paper/2209.11252/) (2022)
- [Knowledge Graph for NLG in the context of conversational agents](https://mlsanity.com/paper/2307.01548/) (2023)
- [TrICy: Trigger-guided Data-to-text Generation with Intent aware Attention-Copy](https://mlsanity.com/paper/2402.01714/) (2024)
- [Enterprise Use Cases Combining Knowledge Graphs and Natural Language Processing](https://mlsanity.com/paper/2404.01443/) (2024)
- [Ta-G-T: Subjectivity Capture in Table to Text Generation via RDF Graphs](https://mlsanity.com/paper/2507.19710/) (2025)

## Abstract (arXiv)

Graph-to-text generation has benefited from pre-trained language models (PLMs) in achieving better performance than structured graph encoders. However, they fail to fully utilize the structure information of the input graph. In this paper, we aim to further improve the performance of the pre-trained language model by proposing a structured graph-to-text model with a two-step fine-tuning mechanism which first fine-tunes the model on Wikipedia before adapting to the graph-to-text generation. In addition to using the traditional token and position embeddings to encode the knowledge graph (KG), we propose a novel tree-level embedding method to capture the inter-dependency structures of the input graph. This new approach has significantly improved the performance of all text generation metrics for the English WebNLG 2017 dataset.