mlsanityData to 7 Oct 2026

CodeGraph: Enhancing Graph Reasoning of LLMs with Code

arXiv 2408.13863 ↗First posted 25 Aug 2024cs.CL cs.AIcited by 6

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

With the increasing popularity of large language models (LLMs), reasoning on basic graph algorithm problems is an essential intermediate step in assessing their abilities to process and infer complex graph reasoning tasks. Existing methods usually convert graph-structured data to textual descriptions and then use LLMs for reasoning and computation. However, LLMs often produce computation errors on arithmetic parts in basic graph algorithm problems, such as counting number of edges. In addition, they struggle to control or understand the output of the reasoning process, raising concerns about whether LLMs are simply guessing. In this paper, we introduce CodeGraph, a method that encodes graph problem solutions as code. The methods solve new graph problems by learning from exemplars, generating programs, and executing them via a program interpreter. Using the few-shot setting, we evaluate CodeGraph with the base LLM being GPT-3.5 Turbo, Llama3-70B Instruct, Mixtral-8x22B Instruct, and Mixtral-8x7B Instruct. Experimental results on six tasks with six graph encoding methods in the GraphQA dataset demonstrate that CodeGraph can boost performance on graph reasoning tasks inside LLMs by 1.3% to 58.6%, depending on the task. Compared to the existing methods, CodeGraph demonstrates strong performance on arithmetic problems in graph tasks and offers a more controllable and interpretable approach to the reasoning process.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 64 of its 100 references that are in the map. First posted on arXiv 25 Aug 2024.

Builds on

64 of 100 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024GraphWiz: An Instruction-Following Language Model for Graph Problems25 Feb
  2. 2023Can Language Models Solve Graph Problems in Natural Language?17 May
  3. 2023Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoning31 Jan

Other references · 61

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2018How Powerful are Graph Neural Networks?cited by 4,355
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  12. 2023Mistral 7Bcited by 3,065
38 more
  1. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  2. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  3. 2024Mixtral of Expertscited by 1,785
  4. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  5. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  6. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  7. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  8. 2023PaLM 2 Technical Reportcited by 1,154
  9. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  10. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  11. 2022PAL: Program-aided Language Modelscited by 635
  12. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  13. 2020Generative Language Modeling for Automated Theorem Provingcited by 379
  14. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  15. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 308
  16. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  17. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 271
  18. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
  19. 2022Binding Language Models in Symbolic Languagescited by 224
  20. 2023GraphGPT: Graph Instruction Tuning for Large Language Modelscited by 223
  21. 2023Language is All a Graph Needscited by 219
  22. 2023Talk like a Graph: Encoding Graphs for Large Language Modelscited by 175
  23. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  24. 2023RestGPT: Connecting Large Language Models with Real-World RESTful APIscited by 129
  25. 2024Let Your Graph Do the Talking: Encoding Structured Data for LLMscited by 112
  26. 2023At Which Training Stage Does Code Data Help LLMs Reasoning?cited by 82
  27. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
  28. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  29. 2022Code4Struct: Code Generation for Few-Shot Event Structure Predictioncited by 70
  30. 2024Understanding Transformer Reasoning Capabilities via Graph Algorithmscited by 64
  31. 2024LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewingcited by 62
  32. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programscited by 45
  33. 2024HiGPT: Heterogeneous Graph Language Modelcited by 44
  34. 2023LLM4DyG: Can Large Language Models Solve Spatial-Temporal Problems on Dynamic Graphs?cited by 40
  35. 2023Evaluating Large Language Models on Graphs: Performance Insights and Comparative Analysiscited by 35
  36. 2023Can LLMs Effectively Leverage Graph Structural Information through Prompts, and Why?cited by 31
  37. 2023COLA: Contextualized Commonsense Causal Reasoning from the Causal Inference Perspectivecited by 22
  38. 2023Advancing Abductive Reasoning in Knowledge Graphs through Complex Logical Hypothesis Generationcited by 19

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025RTL++: Graph-enhanced LLM for RTL Code Generationcited by 16
  2. 2025Graph Counselor: Adaptive Graph Exploration via Multi-Agent Synergy to Enhance LLM Reasoningcited by 6
  3. 2025Actions Speak Louder than Prompts: A Large-Scale Study of LLMs for Graph Inferencecited by 4
  4. 2026GraphSkill: Documentation-Guided Hierarchical Retrieval-Augmented Coding for Complex Graph Reasoningcited by 4
  5. 2026Hypergraph as Languagecited by 0
  6. 2026Unified Multi-Dimensional Benchmark for Complex Graph Reasoning in Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking and Benchmarking Large Language Models for Graph Reasoning
  2. 2024Graph Reasoning with Large Language Models via Pseudo-code Prompting
  3. 2026Are Large Language Models Suitable for Graph Computation? Progress and Prospects
  4. 2025Less is More: Learning Graph Tasks with Just LLMs
  5. 2025Improving LLMs' Generalized Reasoning Abilities by Graph Problems
  6. 2023GraphReason: Enhancing Reasoning Capabilities of Large Language Models through A Graph-Based Verification Approach
  7. 2025Exploring Graph Learning Tasks with Pure LLMs: A Comprehensive Benchmark and Investigation
  8. 2024Can Graph Descriptive Order Affect Solving Graph Problems with LLMs?