# Retrieve-Rewrite-Answer: A KG-to-Text Enhanced LLMs Framework for Knowledge Graph Question Answering

arXiv: https://arxiv.org/abs/2309.11206 · HTML: https://mlsanity.com/paper/2309.11206/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 45 references that are in the map.
- First posted on arXiv 20 Sep 2023.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Object Hallucination in Image Captioning](https://mlsanity.com/paper/1809.02156/) (2018)
- [The Web as a Knowledge-base for Answering Complex Questions](https://mlsanity.com/paper/1803.06643/) (2018)
- [Internet-Augmented Dialogue Generation](https://mlsanity.com/paper/2107.07566/) (2021)
- [Variational Reasoning for Question Answering with Knowledge Graph](https://mlsanity.com/paper/1709.04071/) (2017)
- [HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge](https://mlsanity.com/paper/2304.06975/) (2023)
- [AugGPT: Leveraging ChatGPT for Text Data Augmentation](https://mlsanity.com/paper/2302.13007/) (2023)
- [Text Generation from Knowledge Graphs with Graph Transformers](https://mlsanity.com/paper/1904.02342/) (2019)
- [Investigating Pretrained Language Models for Graph-to-Text Generation](https://mlsanity.com/paper/2007.08426/) (2020)
- [Graph Transformer for Graph-to-Sequence Learning](https://mlsanity.com/paper/1911.07470/) (2019)
- [Sequence-to-Sequence Knowledge Graph Completion and Question Answering](https://mlsanity.com/paper/2203.10321/) (2022)
- [Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answering](https://mlsanity.com/paper/2210.01613/) (2022)
- [Barack's Wife Hillary: Using Knowledge-Graphs for Fact-Aware Language Modeling](https://mlsanity.com/paper/1906.07241/) (2019)
- [BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark](https://mlsanity.com/paper/2302.09432/) (2023)
- [JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs](https://mlsanity.com/paper/2106.10502/) (2021)
- [An Interpretable Reasoning Network for Multi-Relation Question Answering](https://mlsanity.com/paper/1801.04726/) (2018)
- [Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering](https://mlsanity.com/paper/2210.03078/) (2022)
- [Modeling Graph Structure in Transformer for Better AMR-to-Text Generation](https://mlsanity.com/paper/1909.00136/) (2019)
- [Densely Connected Graph Convolutional Networks for Graph-to-Sequence Learning](https://mlsanity.com/paper/1908.05957/) (2019)
- [Deep Graph Convolutional Encoders for Structured Data to Text Generation](https://mlsanity.com/paper/1810.09995/) (2018)

## Built on by (most cited first, 30 of 65)

- [Retrieval-Augmented Generation for AI-Generated Content: A Survey](https://mlsanity.com/paper/2402.19473/) (2024)
- [Graph Retrieval-Augmented Generation: A Survey](https://mlsanity.com/paper/2408.08921/) (2024)
- [ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models](https://mlsanity.com/paper/2404.07738/) (2024)
- [Retrieval-Augmented Generation with Graphs (GraphRAG)](https://mlsanity.com/paper/2501.00309/) (2024)
- [GNN-RAG: Graph Neural Retrieval for Large Language Model Reasoning](https://mlsanity.com/paper/2405.20139/) (2024)
- [Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Survey](https://mlsanity.com/paper/2311.07914/) (2023)
- [Simple Is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation](https://mlsanity.com/paper/2410.20724/) (2024)
- [Knowledge Graph Retrieval-Augmented Generation for LLM-based Recommendation](https://mlsanity.com/paper/2501.02226/) (2025)
- [Multimodal Reasoning with Multimodal Knowledge Graph](https://mlsanity.com/paper/2406.02030/) (2024)
- [A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directions](https://mlsanity.com/paper/2406.03712/) (2024)
- [Graph Machine Learning in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2404.14928/) (2024)
- [Can Knowledge Graphs Make Large Language Models More Trustworthy? An Empirical Study Over Open-ended Question Answering](https://mlsanity.com/paper/2410.08085/) (2024)
- [T-RAG: Lessons from the LLM Trenches](https://mlsanity.com/paper/2402.07483/) (2024)
- [KnowledgeNavigator: Leveraging Large Language Models for Enhanced Reasoning over Knowledge Graph](https://mlsanity.com/paper/2312.15880/) (2023)
- [Decoding on Graphs: Faithful and Sound Reasoning on Knowledge Graphs through Generation of Well-Formed Chains](https://mlsanity.com/paper/2410.18415/) (2024)
- [Aligning Vision to Language: Annotation-Free Multimodal Knowledge Graph Construction for Enhanced LLMs Reasoning](https://mlsanity.com/paper/2503.12972/) (2025)
- [Two-stage Generative Question Answering on Temporal Knowledge Graph Using Large Language Models](https://mlsanity.com/paper/2402.16568/) (2024)
- [GraphOTTER: Evolving LLM-based Graph Reasoning for Complex Table Question Answering](https://mlsanity.com/paper/2412.01230/) (2024)
- [Graph-based Approaches and Functionalities in Retrieval-Augmented Generation: A Comprehensive Survey](https://mlsanity.com/paper/2504.10499/) (2025)
- [Knowledge Graph-extended Retrieval Augmented Generation for Question Answering](https://mlsanity.com/paper/2504.08893/) (2025)
- [Mindful-RAG: A Study of Points of Failure in Retrieval Augmented Generation](https://mlsanity.com/paper/2407.12216/) (2024)
- [Graph2text or Graph2token: A Perspective of Large Language Models for Graph Learning](https://mlsanity.com/paper/2501.01124/) (2025)
- [Large Language Models Can Better Understand Knowledge Graphs Than We Thought](https://mlsanity.com/paper/2402.11541/) (2024)
- [Evidence-Focused Fact Summarization for Knowledge-Augmented Zero-Shot Question Answering](https://mlsanity.com/paper/2403.02966/) (2024)
- [LightPROF: A Lightweight Reasoning Framework for Large Language Model on Knowledge Graph](https://mlsanity.com/paper/2504.03137/) (2025)
- [EPERM: An Evidence Path Enhanced Reasoning Model for Knowledge Graph Question and Answering](https://mlsanity.com/paper/2502.16171/) (2025)
- [Way to Specialist: Closing Loop Between Specialized LLM and Evolving Domain Knowledge Graph](https://mlsanity.com/paper/2411.19064/) (2024)
- [Beyond Chunks and Graphs: Retrieval-Augmented Generation through Triplet-Driven Thinking](https://mlsanity.com/paper/2508.02435/) (2025)
- [Reasoning by Exploration: A Unified Approach to Retrieval and Generation over Graphs](https://mlsanity.com/paper/2510.07484/) (2025)
- [Foundation Models for CPS-IoT: Opportunities and Challenges](https://mlsanity.com/paper/2501.16368/) (2025)

## Abstract (arXiv)

Despite their competitive performance on knowledge-intensive tasks, large language models (LLMs) still have limitations in memorizing all world knowledge especially long tail knowledge. In this paper, we study the KG-augmented language model approach for solving the knowledge graph question answering (KGQA) task that requires rich world knowledge. Existing work has shown that retrieving KG knowledge to enhance LLMs prompting can significantly improve LLMs performance in KGQA. However, their approaches lack a well-formed verbalization of KG knowledge, i.e., they ignore the gap between KG representations and textual representations. To this end, we propose an answer-sensitive KG-to-Text approach that can transform KG knowledge into well-textualized statements most informative for KGQA. Based on this approach, we propose a KG-to-Text enhanced LLMs framework for solving the KGQA task. Experiments on several KGQA benchmarks show that the proposed KG-to-Text augmented LLMs approach outperforms previous KG-augmented LLMs approaches regarding answer accuracy and usefulness of knowledge statements.