Revisiting Graph-Tokenizing Large Language Models: A Systematic Evaluation of Graph Token Understanding
Not on the map. Why?
Abstract (arXiv)
The remarkable success of large language models (LLMs) has motivated researchers to adapt them as universal predictors for various graph tasks. As a widely recognized paradigm, Graph-Tokenizing LLMs (GTokenLLMs) compress complex graph data into graph tokens and treat them as prefix tokens for querying LLMs, leading many to believe that LLMs can understand graphs more effectively and efficiently. In this paper, we challenge this belief: textitDo GTokenLLMs fully understand graph tokens in the natural-language embedding space? Motivated by this question, we formalize a unified framework for GTokenLLMs and propose an evaluation pipeline, GTEval, to assess graph-token understanding via instruction transformations at the format and content levels. We conduct extensive experiments on 6 representative GTokenLLMs with GTEval. The primary findings are as follows: (1) Existing GTokenLLMs do not fully understand graph tokens. They exhibit over-sensitivity or over-insensitivity to instruction changes, and rely heavily on text for reasoning; (2) Although graph tokens preserve task-relevant graph information and receive attention across LLM layers, their utilization varies across models and instruction variants; (3) Additional instruction tuning can improve performance on the original and seen instructions, but it does not fully address the challenge of graph-token understanding, calling for further improvement.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 39 of its 48 references that are in the map. First posted on arXiv 5 May 2026.
Builds on
39 of 48 references.
References · 39
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2017Inductive Representation Learning on Large Graphscited by 6,284
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
- 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
- 2016Revisiting Semi-Supervised Learning with Graph Embeddingscited by 970
- 2023Vision Transformers Need Registerscited by 844
- 2020A Generalization of Transformer Networks to Graphscited by 653
27 more
- 2023Instruction Tuning for Large Language Models: A Surveycited by 617
- 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 271
- 2020Wiki-CS: A Wikipedia-Based Benchmark for Graph Neural Networkscited by 253
- 2023Can Language Models Solve Graph Problems in Natural Language?cited by 249
- 2023GraphGPT: Graph Instruction Tuning for Large Language Modelscited by 223
- 2023Language is All a Graph Needscited by 219
- 2023GPT4Graph: Can Large Language Models Understand Graph Structured Data ? An Empirical Evaluation and Benchmarkingcited by 209
- 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 190
- 2024LLaGA: Large Language and Graph Assistantcited by 177
- 2023Talk like a Graph: Encoding Graphs for Large Language Modelscited by 175
- 2023GraphText: Graph Reasoning in Text Spacecited by 110
- 2023Graph Foundation Models: Concepts, Opportunities and Challengescited by 101
- 2021Robustness of Graph Neural Networks at Scalecited by 100
- 2023SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learningcited by 73
- 2023Label-free Node Classification on Graphs with Large Language Models (LLMS)cited by 72
- 2024GraphTranslator: Aligning Graph Model to Large Language Model for Open-ended Taskscited by 63
- 2024LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddingscited by 60
- 2024InstructGraph: Boosting Large Language Models via Graph-centric Instruction Tuning and Preference Alignmentcited by 56
- 2024GOFA: A Generative One-For-All Model for Joint Graph Language Modelingcited by 56
- 2023Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Taskscited by 53
- 2024How Do Large Language Models Understand Graph Patterns? A Benchmark for Graph Pattern Comprehensioncited by 30
- 2024Can Large Language Models Improve the Adversarial Robustness of Graph Neural Networks?cited by 22
- 2024Graph-oriented Instruction Tuning of Large Language Models for Generic Graph Miningcited by 20
- 2024Can Large Language Models Analyze Graphs like Professionals? A Benchmark, Datasets and Modelscited by 17
- 2024Learning on Graphs with Large Language Models(LLMs): A Deep Dive into Model Robustnesscited by 11
- 2025UniGTE: Unified Graph-Text Encoding for Zero-Shot Generalization across Graph Tasks and Domainscited by 7
- 2024From Anchors to Answers: A Novel Node Tokenizer for Integrating Graph Structure into Large Language Modelscited by 4
Built on by
Citing papers per year: 2026: 2.
2 papers.