mlsanityData to 14 Oct 2026

Can LLMs Effectively Leverage Graph Structural Information through Prompts, and Why?

arXiv 2309.16595 ↗First posted 28 Sep 2023cs.LG cs.AIcited by 31

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are gaining increasing attention for their capability to process graphs with rich text attributes, especially in a zero-shot fashion. Recent studies demonstrate that LLMs obtain decent text classification performance on common text-rich graph benchmarks, and the performance can be improved by appending encoded structural information as natural languages into prompts. We aim to understand why the incorporation of structural information inherent in graph data can improve the prediction performance of LLMs. First, we rule out the concern of data leakage by curating a novel leakage-free dataset and conducting a comparative analysis alongside a previously widely-used dataset. Second, as past work usually encodes the ego-graph by describing the graph structure in natural language, we ask the question: do LLMs understand the graph structure in accordance with the intent of the prompt designers? Third, we investigate why LLMs can improve their performance after incorporating structural information. Our exploration of these questions reveals that (i) there is no substantial evidence that the performance of LLMs is significantly attributed to data leakage; (ii) instead of understanding prompts as graph structures as intended by the prompt designers, LLMs tend to process prompts more as contextual paragraphs and (iii) the most efficient elements of the local neighborhood included in the prompt are phrases that are pertinent to the node label, rather than the graph structure.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 27 of its 48 references that are in the map. First posted on arXiv 28 Sep 2023.

Builds on

27 of 48 references.

References · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  5. 2017Graph Attention Networkscited by 8,923
  6. 2017Inductive Representation Learning on Large Graphscited by 6,320
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  8. 2017Neural Message Passing for Quantum Chemistrycited by 3,487
  9. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  10. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,434
  11. 2020Extracting Training Data from Large Language Modelscited by 2,189
  12. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
15 more
  1. 2016Revisiting Semi-Supervised Learning with Graph Embeddingscited by 976
  2. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  3. 2021Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methodscited by 311
  4. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 274
  5. 2023Can Language Models Solve Graph Problems in Natural Language?cited by 252
  6. 2023Language is All a Graph Needscited by 220
  7. 2023GPT4Graph: Can Large Language Models Understand Graph Structured Data ? An Empirical Evaluation and Benchmarkingcited by 210
  8. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 200
  9. 2019On the Use of ArXiv as a Datasetcited by 103
  10. 2023Can we trust the evaluation on ChatGPT?cited by 63
  11. 2021Improving Graph Neural Networks with Simple Architecture Designcited by 41
  12. 2020Grale: Designing Networks for Graph Learningcited by 28
  13. 2015Homophily and missing links in citation networkscited by 21
  14. 2023On Performance Discrepancies Across Local Homophily Levels in Graph Neural Networkscited by 13
  15. 2022Graph Learning Indexer: A Contributor-Friendly and Metadata-Rich Platform for Graph Learning Benchmarkscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 13; 2026: 9.

31 papers.

Sort:

By area:

  1. 2025A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Modelscited by 88
  2. 2024Demystifying Chains, Trees, and Graphs of Thoughtscited by 63
  3. 2024How Do Large Language Models Understand Graph Patterns? A Benchmark for Graph Pattern Comprehensioncited by 30
  4. 2025When Do LLMs Help With Node Classification? A Comprehensive Analysiscited by 21
  5. 2024A Survey of Large Language Models on Generative Graph Analytics: Query, Learning, and Applicationscited by 17
  6. 2024CATP-LLM: Empowering Large Language Models for Cost-Aware Tool Planningcited by 14
  7. 2025SeqGrowGraph: Learning Lane Topology as a Chain of Graph Expansionscited by 10
  8. 2024A Survey on Self-Supervised Graph Foundation Models: Knowledge-Based Perspectivecited by 9
  9. 2024Hybrid LLM/Rule-based Approaches to Business Insights Generation from Structured Datacited by 7
  10. 2025What Makes a Good Natural Language Prompt?cited by 7
  11. 2026GraphSearch: Agentic Search-Augmented Reasoning for Zero-Shot Graph Learningcited by 7
  12. 2024CodeGraph: Enhancing Graph Reasoning of LLMs with Codecited by 6
19 more
  1. 2025Graph-R1: Incentivizing the Zero-Shot Graph Learning Capability in LLMs via Explicit Reasoningcited by 6
  2. 2025Large Language Models Meet Text-Attributed Graphs: A Survey of Integration Frameworks and Applicationscited by 6
  3. 2025Actions Speak Louder than Prompts: A Large-Scale Study of LLMs for Graph Inferencecited by 5
  4. 2026GNN-as-Judge: Unleashing the Power of LLMs for Graph Learning with GNN Feedbackcited by 4
  5. 2024Integrating Large Language Models with Graph-based Reasoning for Conversational Question Answeringcited by 3
  6. 2024A Hierarchical Language Model For Interpretable Graph Reasoningcited by 3
  7. 2025Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More Morecited by 3
  8. 2025Court of LLMs: Evidence-Augmented Generation via Multi-LLM Collaboration for Text-Attributed Graph Anomaly Detectioncited by 3
  9. 2025TinyGraphEstimator: Adapting Lightweight Language Models for Graph Structure Inferencecited by 3
  10. 2025When Structure Doesn't Help: LLMs Do Not Read Text-Attributed Graphs as Effectively as We Expectedcited by 3
  11. 2026Mario: Multimodal Graph Reasoning with Large Language Modelscited by 3
  12. 2025Detecting Hallucinations in Graph Retrieval-Augmented Generation via Attention Patterns and Semantic Alignmentcited by 1
  13. 2026Clustering as Reasoning: A k-Means Interpretation of Chain-of-Thought Graph Learningcited by 1
  14. 2025Dual Refinement Cycle Learning: Unsupervised Text Classification of Mamba and Community Detection on Text Attributed Graphcited by 0
  15. 2026From Perception to Action: Spatial AI Agents and World Modelscited by 0
  16. 2026Knowledge Graph-Enhanced Zero-Shot Topic Classification: A Multi-Strategy Comparative Studycited by 0
  17. 2026Beyond the Golden Teacher: Enhancing Graph Learning through LLM-GNN Co-teachingcited by 0
  18. 2026GLIP: Graph and LLM Joint Pretraining for Graph-Level Taskscited by 0
  19. 2026Agentic Graph Token Reasoningcited by 0