mlsanityData to 7 Oct 2026

StructGPT: A General Framework for Large Language Model to Reason over Structured Data

arXiv 2305.09645 ↗First posted 16 May 2023cs.CLcited by 339

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

In this paper, we study how to improve the zero-shot reasoning ability of large language models~(LLMs) over structured data in a unified way. Inspired by the study on tool augmentation for LLMs, we develop an Iterative Reading-then-Reasoning~(IRR) approach for solving question answering tasks based on structured data, called StructGPT. In our approach, we construct the specialized function to collect relevant evidence from structured data (ie reading), and let LLMs concentrate the reasoning task based on the collected information (ie reasoning). Specially, we propose an invoking-linearization-generation procedure to support LLMs in reasoning on the structured data with the help of the external interfaces. By iterating this procedures with provided interfaces, our approach can gradually approach the target answer to a given query. Extensive experiments conducted on three types of structured data demonstrate the effectiveness of our approach, which can significantly boost the performance of ChatGPT and achieve comparable performance against the full-data supervised-tuning baselines. Our codes and data are publicly available at~urlhttps://github.com/RUCAIBox/StructGPT.

Key facts. Cited by 339 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 47 of its 57 references that are in the map. First posted on arXiv 16 May 2023.

Builds on

47 of 57 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity8 Feb
  2. 2022PAL: Program-aided Language Models18 Nov
  3. 2022Great Truths are Always Simple: A Rather Simple Knowledge Encoder for Enhancing the Commonsense Reasoning Capacity of Pre-Trained Models4 May

Other references · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  10. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  11. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  12. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
32 more
  1. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  2. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  3. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 432
  4. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  5. 2019RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parserscited by 420
  6. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  7. 2021TAPEX: Table Pre-training via Learning a Neural SQL Executorcited by 233
  8. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  9. 2022Binding Language Models in Symbolic Languagescited by 224
  10. 2016Key-Value Memory Networks for Directly Reading Documentscited by 223
  11. 2023Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoningcited by 177
  12. 2017Variational Reasoning for Question Answering with Knowledge Graphcited by 170
  13. 2018Open Domain Question Answering Using Early Fusion of Knowledge Bases and Textcited by 167
  14. 2020Structure-Grounded Pretraining for Text-to-SQLcited by 126
  15. 2023A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capabilitycited by 118
  16. 2022Large Language Models are few(1)-shot Table Reasonerscited by 116
  17. 2021Towards Robustness of Text-to-SQL Models against Synonym Substitutioncited by 110
  18. 2021Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signalscited by 100
  19. 2022UniKGQA: Unified Retrieval and Reasoning for Solving Multi-hop Question Answering Over Knowledge Graphcited by 100
  20. 2022RASAT: Integrating Relational Structures into Pretrained Seq2Seq Model for Text-to-SQLcited by 85
  21. 2021A Survey on Complex Knowledge Base Question Answering: Methods, Challenges and Solutionscited by 77
  22. 2020Efficient One-Pass End-to-End Entity Linking for Questionscited by 76
  23. 2020Understanding tables with intermediate pre-trainingcited by 68
  24. 2020UniK-QA: Unified Representations of Structured and Unstructured Knowledge for Open-Domain Question Answeringcited by 61
  25. 2018Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsingcited by 57
  26. 2022Don't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environmentscited by 57
  27. 2022Transcending Scaling Laws with 0.1% Extra Computecited by 55
  28. 2021Knowledge Enhanced Pretrained Language Models: A Compreshensive Surveycited by 33
  29. 2022SKILL: Structured Knowledge Infusion for Large Language Modelscited by 27
  30. 2023From Zero to Hero: Examining the Power of Symbolic Tasks in Instruction Tuningcited by 14
  31. 2022Towards Generalizable and Robust Text-to-SQL Parsingcited by 7
  32. 2022Injecting Domain Knowledge in Language Models for Task-Oriented Dialogue Systemscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 32; 2024: 119; 2025: 118; 2026: 70.

339 papers.

Sort:

By area:

  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  2. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  3. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  4. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  5. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  6. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 329
  7. 2024Graph Retrieval-Augmented Generation: A Surveycited by 320
  8. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 273
  9. 2023Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graphcited by 235
  10. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  11. 2023Language is All a Graph Needscited by 219
  12. 2024G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answeringcited by 217
38 more
  1. 2023GPT4Graph: Can Large Language Models Understand Graph Structured Data ? An Empirical Evaluation and Benchmarkingcited by 209
  2. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  3. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 190
  4. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 183
  5. 2023MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQLcited by 158
  6. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 157
  7. 2024AutoWebGLM: A Large Language Model-based Web Navigating Agentcited by 130
  8. 2024GNN-RAG: Graph Neural Retrieval for Large Language Model Reasoningcited by 128
  9. 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 118
  10. 2024Large Language Models Meet NLP: A Surveycited by 102
  11. 2024AriGraph: Learning Knowledge Graph World Models with Episodic Memory for LLM Agentscited by 100
  12. 2023Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Surveycited by 92
  13. 2024KG-Agent: An Efficient Autonomous Agent Framework for Complex Reasoning over Knowledge Graphcited by 90
  14. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  15. 2024Simple Is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generationcited by 78
  16. 2024GRAG: Graph Retrieval-Augmented Generationcited by 77
  17. 2024Plan-on-Graph: Self-Correcting Adaptive Planning of Large Language Model on Knowledge Graphscited by 77
  18. 2023Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflowcited by 76
  19. 2024TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarioscited by 76
  20. 2024KAG: Boosting LLMs in Professional Domains via Knowledge Augmented Generationcited by 67
  21. 2024Generate-on-Graph: Treat LLM as both Agent and KG in Incomplete Knowledge Graph Question Answeringcited by 59
  22. 2024InstructGraph: Boosting Large Language Models via Graph-centric Instruction Tuning and Preference Alignmentcited by 56
  23. 2024KG-RAG: Bridging the Gap Between Knowledge and Creativitycited by 55
  24. 2024Large Language Model for Table Processing: A Surveycited by 52
  25. 2024DALK: Dynamic Co-Augmentation of LLMs and KG to answer Alzheimer's Disease Questions with Scientific Literaturecited by 50
  26. 2023Evaluating and Improving Tool-Augmented Computation-Intensive Math Reasoningcited by 49
  27. 2025DeepAnalyze: Agentic Large Language Models for Autonomous Data Sciencecited by 49
  28. 2023Making Large Language Models Perform Better in Knowledge Graph Completioncited by 48
  29. 2024Graph-constrained Reasoning: Faithful Reasoning on Knowledge Graphs with Large Language Modelscited by 47
  30. 2024LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generationcited by 45
  31. 2024MegaAgent: A Large-Scale Autonomous LLM-based Multi-Agent System Without Predefined SOPscited by 45
  32. 2025In-depth Analysis of Graph-based RAG in a Unified Frameworkcited by 45
  33. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 45
  34. 2023ChatKBQA: A Generate-then-Retrieve Framework for Knowledge Base Question Answering with Fine-tuned Large Language Modelscited by 44
  35. 2024Paths-over-Graph: Knowledge Graph Empowered Large Language Model Reasoningcited by 44
  36. 2023ChatCoT: Tool-Augmented Chain-of-Thought Reasoning on Chat-based Large Language Modelscited by 43
  37. 2023OceanGPT: A Large Language Model for Ocean Science Taskscited by 42
  38. 2024PlanRAG: A Plan-then-Retrieval Augmented Generation for Generative Large Language Models as Decision Makerscited by 42

50 most cited of 339 All 339 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026π2: Structure-Originated Reasoning Data Improves Long-Context Reasoning Ability of Large Language Models
  2. 2024StruEdit: Structured Outputs Enable the Fast and Accurate Knowledge Editing for Large Language Models
  3. 2024Reasoning Factual Knowledge in Structured Data with Large Language Models
  4. 2024StrucText-Eval: Evaluating Large Language Model's Reasoning Ability in Structure-Rich Text
  5. 2024TAT-LLM: A Specialized Language Model for Discrete Reasoning over Tabular and Textual Data
  6. 2025Enhancing Large Language Models through Structured Reasoning
  7. 2026From Chains to Graphs: Self-Structured Reasoning for General-Domain LLMs
  8. 2023Empirical Study of Zero-Shot NER with ChatGPT