# Evaluating and Improving Graph to Text Generation with Large Language Models

arXiv: https://arxiv.org/abs/2501.14497 · HTML: https://mlsanity.com/paper/2501.14497/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 66 references that are in the map.
- First posted on arXiv 24 Jan 2025.

## Builds on (most cited first, 29 of 29)

- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [DropEdge: Towards Deep Graph Convolutional Networks on Node Classification](https://mlsanity.com/paper/1907.10903/) (2019)
- [Compositional Semantic Parsing on Semi-Structured Tables](https://mlsanity.com/paper/1508.00305/) (2015)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [A Fair Comparison of Graph Neural Networks for Graph Classification](https://mlsanity.com/paper/1912.09893/) (2019)
- [DART: Open-Domain Structured Data Record to Text Generation](https://mlsanity.com/paper/2007.02871/) (2020)
- [Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Framework](https://mlsanity.com/paper/2305.03268/) (2023)
- [Text Generation from Knowledge Graphs with Graph Transformers](https://mlsanity.com/paper/1904.02342/) (2019)
- [Investigating Pretrained Language Models for Graph-to-Text Generation](https://mlsanity.com/paper/2007.08426/) (2020)
- [Data-to-Text Generation with Content Selection and Planning](https://mlsanity.com/paper/1809.00582/) (2018)
- [Dr.ICL: Demonstration-Retrieved In-context Learning](https://mlsanity.com/paper/2305.14128/) (2023)
- [SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphs](https://mlsanity.com/paper/2308.03349/) (2023)
- [Modeling Global and Local Node Contexts for Text Generation from Knowledge Graphs](https://mlsanity.com/paper/2001.11003/) (2020)
- [Densely Connected Graph Convolutional Networks for Graph-to-Sequence Learning](https://mlsanity.com/paper/1908.05957/) (2019)
- [Few-shot Knowledge Graph-to-Text Generation with Pretrained Language Models](https://mlsanity.com/paper/2106.01623/) (2021)
- [Findings of the E2E NLG Challenge](https://mlsanity.com/paper/1810.01170/) (2018)
- [Commonsense Properties from Query Logs and Question Answering Forums](https://mlsanity.com/paper/1905.10989/) (2019)
- [TrustScore: Reference-Free Evaluation of LLM Response Trustworthiness](https://mlsanity.com/paper/2402.12545/) (2024)
- [CoTKR: Chain-of-Thought Enhanced Knowledge Rewriting for Complex Knowledge Graph Question Answering](https://mlsanity.com/paper/2409.19753/) (2024)
- [Evaluating Generative Models for Graph-to-Text Generation](https://mlsanity.com/paper/2307.14712/) (2023)
- [Knowledge Graph Generation From Text](https://mlsanity.com/paper/2211.10511/) (2022)
- [Promoting Graph Awareness in Linearized Graph-to-Text Generation](https://mlsanity.com/paper/2012.15793/) (2020)
- [Transformer-based Entity Typing in Knowledge Graphs](https://mlsanity.com/paper/2210.11151/) (2022)
- [UniArk: Improving Generalisation and Consistency for Factual Knowledge Extraction through Debiasing](https://mlsanity.com/paper/2404.01253/) (2024)
- [Syntax Controlled Knowledge Graph-to-Text Generation with Order and Semantic Consistency](https://mlsanity.com/paper/2207.00719/) (2022)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated immense potential across various tasks. However, research for exploring and improving the capabilities of LLMs in interpreting graph structures remains limited. To address this gap, we conduct a comprehensive evaluation of prompting current open-source LLMs on graph-to-text generation tasks. Although we explored the optimal prompting strategies and proposed a novel and effective diversity-difficulty-based few-shot sample selection method, we found that the improvements from tuning-free approaches were incremental, as LLMs struggle with planning on complex graphs, particularly those with a larger number of triplets. To further improve LLMs in planning with graph sequences and grounding in truth, we introduce a new graph-to-text dataset, PlanGTG, annotated with two sub-tasks: reordering and attribution. Through extensive automatic and human evaluations, we demonstrate significant improvements in the quality of generated text from both few-shot learning and fine-tuning perspectives using the PlanGTG dataset. Our study paves the way for new research directions in graph-to-text generation. PlanGTG datasets can be found in https://github.com/probe2/kg_text.