# GraphextQA: A Benchmark for Evaluating Graph-Enhanced Large Language Models

arXiv: https://arxiv.org/abs/2310.08487 · HTML: https://mlsanity.com/paper/2310.08487/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 12 Oct 2023.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [The Web as a Knowledge-base for Answering Complex Questions](https://mlsanity.com/paper/1803.06643/) (2018)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Text](https://mlsanity.com/paper/1904.09537/) (2019)
- [Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases](https://mlsanity.com/paper/2011.07743/) (2020)
- [Deep Bidirectional Language-Knowledge Graph Pretraining](https://mlsanity.com/paper/2210.09338/) (2022)
- [MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering](https://mlsanity.com/paper/2007.15207/) (2020)
- [Entities as Experts: Sparse Memory Access with Entity Supervision](https://mlsanity.com/paper/2004.07202/) (2020)
- [Subgraph Retrieval Enhanced Model for Multi-hop Knowledge Base Question Answering](https://mlsanity.com/paper/2202.13296/) (2022)
- [KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Base](https://mlsanity.com/paper/2007.03875/) (2020)
- [A Survey on Complex Knowledge Base Question Answering: Methods, Challenges and Solutions](https://mlsanity.com/paper/2105.11644/) (2021)
- [Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answering](https://mlsanity.com/paper/2210.01613/) (2022)
- [Grounded Conversation Generation as Guided Traverses in Commonsense Knowledge Graphs](https://mlsanity.com/paper/1911.02707/) (2019)
- [GraphVite: A High-Performance CPU-GPU Hybrid System for Node Embedding](https://mlsanity.com/paper/1903.00757/) (2019)
- [QALD-9-plus: A Multilingual Dataset for Question Answering over DBpedia and Wikidata Translated by Native Speakers](https://mlsanity.com/paper/2202.00120/) (2022)
- [Graph Reasoning for Question Answering with Triplet Retrieval](https://mlsanity.com/paper/2305.18742/) (2023)
- [Compositional Generalization in Multilingual Semantic Parsing over Wikidata](https://mlsanity.com/paper/2108.03509/) (2021)

## Built on by (most cited first, 2 of 2)

- [A Survey of Large Language Models on Generative Graph Analytics: Query, Learning, and Applications](https://mlsanity.com/paper/2404.14809/) (2024)
- [The Graph Language: How Knowledge Graphs Speak to Large Language Models](https://mlsanity.com/paper/2608.01175/) (2026)

## Abstract (arXiv)

While multi-modal models have successfully integrated information from image, video, and audio modalities, integrating graph modality into large language models (LLMs) remains unexplored. This discrepancy largely stems from the inherent divergence between structured graph data and unstructured text data. Incorporating graph knowledge provides a reliable source of information, enabling potential solutions to address issues in text generation, e.g., hallucination, and lack of domain knowledge. To evaluate the integration of graph knowledge into language models, a dedicated dataset is needed. However, there is currently no benchmark dataset specifically designed for multimodal graph-language models. To address this gap, we propose GraphextQA, a question answering dataset with paired subgraphs, retrieved from Wikidata, to facilitate the evaluation and future development of graph-language models. Additionally, we introduce a baseline model called CrossGNN, which conditions answer generation on the paired graphs by cross-attending question-aware graph features at decoding. The proposed dataset is designed to evaluate graph-language models' ability to understand graphs and make use of it for answer generation. We perform experiments with language-only models and the proposed graph-language model to validate the usefulness of the paired graphs and to demonstrate the difficulty of the task.