mlsanityData to 7 Oct 2026

Call Me When Necessary: LLMs can Efficiently and Faithfully Reason over Structured Environments

arXiv 2403.08593 ↗First posted 13 Mar 2024cs.CL cs.AIcited by 27

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

Large Language Models (LLMs) have shown potential in reasoning over structured environments, e.g., knowledge graph and table. Such tasks typically require multi-hop reasoning, i.e., match natural language utterance with instances in the environment. Previous methods leverage LLMs to incrementally build a reasoning path, where the LLMs either invoke tools or pick up schemas by step-by-step interacting with the environment. We propose Reasoning-Path-Editing (Readi), a novel framework where LLMs can efficiently and faithfully reason over structured environments. In Readi, LLMs initially generate a reasoning path given a query, and edit the path only when necessary. We instantiate the path on structured environments and provide feedback to edit the path if anything goes wrong. Experimental results on three KGQA and two TableQA datasets show the effectiveness of Readi, significantly surpassing previous LLM-based methods (by 9.1% Hit@1 on WebQSP, 12.4% on MQA-3H and 9.5% on WTQ), comparable with state-of-the-art fine-tuned methods (67% on CWQ and 74.7% on WebQSP) and substantially boosting the vanilla LLMs (by 14.9% on CWQ). Our code will be available on https://aka.ms/readi.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 13 of its 50 references that are in the map. First posted on arXiv 13 Mar 2024.

Builds on

13 of 50 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Large Language Models Can Learn Temporal Reasoning12 Jan
  2. 2023Rethinking Tabular Data Understanding with Large Language Models27 Dec

Other references · 11

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023Teaching Large Language Models to Self-Debugcited by 963
  5. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  6. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  7. 2023Tool Learning with Foundation Modelscited by 302
  8. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  9. 2022Holistic Evaluation of Language Modelscited by 115
  10. 2023TaskWeaver: A Code-First Agent Frameworkcited by 85
  11. 2023Can ChatGPT Replace Traditional KBQA Models? An In-depth Analysis of the Question Answering Performance of the GPT LLM Familycited by 71

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 14; 2026: 5.

27 papers.

Sort:

By area:

  1. 2025A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Modelscited by 85
  2. 2024A Survey of Large Language Models for Graphscited by 71
  3. 2024Generate-on-Graph: Treat LLM as both Agent and KG in Incomplete Knowledge Graph Question Answeringcited by 59
  4. 2024QueryAgent: A Reliable and Efficient Reasoning Framework with Environmental Feedback-based Self-Correctioncited by 27
  5. 2024Understanding the Interplay between Parametric and Contextual Knowledge for Large Language Modelscited by 14
  6. 2024TrustUQA: A Trustful Framework for Unified Structured Data Question Answeringcited by 12
  7. 2025BioMaze: Benchmarking and Enhancing Large Language Models for Biological Pathway Reasoningcited by 10
  8. 2024Temporal Knowledge Graph Question Answering: A Surveycited by 9
  9. 2025Atomic Skills are the Prerequisite: When Reinforcement Learning Synthesizes Compositional Reasoning, and When It Only Amplifiescited by 8
  10. 2025Reason-Align-Respond: Aligning LLM Reasoning with Knowledge Graphs for KGQAcited by 7
  11. 2024TARGA: Targeted Synthetic Data Generation for Practical Reasoning over Structured Datacited by 5
  12. 2025Search-on-Graph: Iterative Informed Navigation for Large Language Model Reasoning on Knowledge Graphscited by 5
15 more
  1. 2024From Anchors to Answers: A Novel Node Tokenizer for Integrating Graph Structure into Large Language Modelscited by 4
  2. 2025Self-Reflective Planning with Knowledge Graphs: Enhancing LLM Reasoning Reliability for Question Answeringcited by 4
  3. 2025Chain-of-Query: Unleashing the Power of LLMs in SQL-Aided Table Understanding via Multi-Agent Collaborationcited by 4
  4. 2025Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluationcited by 4
  5. 2025Know3-RAG: A Knowledge-aware RAG Framework with Adaptive Retrieval, Generation, and Filteringcited by 2
  6. 2025Plan of Knowledge: Retrieval-Augmented Large Language Models for Temporal Knowledge Graph Question Answeringcited by 2
  7. 2025Self-Correction Distillation for Structured Data Question Answeringcited by 2
  8. 2026Towards Automated Air Traffic Safety Assessment Around Non-Towered Airports Using Large Language Modelscited by 2
  9. 2026Code-on-Graph: Iterative Programmatic Reasoning via Large Language Models on Knowledge Graphscited by 2
  10. 2025Pandora: Leveraging Code-driven Knowledge Transfer for Unified Structured Knowledge Reasoningcited by 1
  11. 2025The Role of Exploration Modules in Small Language Models for Knowledge Graph Question Answeringcited by 1
  12. 2025Differentiable Evolutionary Reinforcement Learningcited by 1
  13. 2026Runtime Burden Allocation for Structured LLM Routing in Agentic Expert Systems: A Full-Factorial Cross-Backend Methodologycited by 0
  14. 2026CRAFTQA: A Code-Driven Adaptive Framework for Complex Structured Data Reasoningcited by 0
  15. 2026X-Tree: Tokenizing Reusable Experience for Efficient Agent Generalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Knowledge Editing for Multi-Hop Question Answering Using Semantic Analysis
  2. 2025From Implicit Exploration to Structured Reasoning: Leveraging Guideline and Refinement for LLMs
  3. 2025Reason-KE++: Aligning the Process, Not Just the Outcome, for Faithful LLM Knowledge Editing
  4. 2022Large Language Models are few(1)-shot Table Reasoners
  5. 2024Reasoning on Efficient Knowledge Paths:Knowledge Graph Guides Large Language Model for Domain Question Answering
  6. 2024Thinking with Knowledge Graphs: Enhancing LLM Reasoning Through Structured Data
  7. 2024Eliciting Better Multilingual Structured Reasoning from LLMs through Code
  8. 2025Last Layer Logits to Logic: Empowering LLMs with Logic-Consistent Structured Knowledge Reasoning