mlsanityData to 7 Oct 2026

Learning to Plan for Retrieval-Augmented Large Language Models from Knowledge Graphs

arXiv 2406.14282 ↗First posted 20 Jun 2024cs.CL cs.AIcited by 24

On the map: Retrieval-augmented generation → Knowledge-graph and structured-knowledge RAG

Abstract (arXiv)

Improving the performance of large language models (LLMs) in complex question-answering (QA) scenarios has always been a research focal point. Recent studies have attempted to enhance LLMs' performance by combining step-wise planning with external retrieval. While effective for advanced models like GPT-3.5, smaller LLMs face challenges in decomposing complex questions, necessitating supervised fine-tuning. Previous work has relied on manual annotation and knowledge distillation from teacher LLMs, which are time-consuming and not accurate enough. In this paper, we introduce a novel framework for enhancing LLMs' planning capabilities by using planning data derived from knowledge graphs (KGs). LLMs fine-tuned with this data have improved planning capabilities, better equipping them to handle complex QA tasks that involve retrieval. Evaluations on multiple datasets, including our newly proposed benchmark, highlight the effectiveness of our framework and the benefits of KG-derived planning data.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Knowledge-graph and structured-knowledge RAG, one of 15 directions in Retrieval-augmented generation. Builds on 33 of its 53 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

33 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy24 May

Other references · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  10. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  11. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  12. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
20 more
  1. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  2. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  3. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  4. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  5. 2023Large Language Modelscited by 774
  6. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  7. 2023Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoningcited by 326
  8. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  9. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 194
  10. 2020Query2box: Reasoning over Knowledge Graphs in Vector Space using Box Embeddingscited by 169
  11. 2024RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generationcited by 144
  12. 2020Beta Embeddings for Multi-Hop Logical Reasoning in Knowledge Graphscited by 128
  13. 2023Chain-of-Knowledge: Grounding Large Language Models via Dynamic Knowledge Adapting over Heterogeneous Sourcescited by 121
  14. 2018Embedding Logical Queries on Knowledge Graphscited by 120
  15. 2024Large Language Models Can Learn Temporal Reasoningcited by 110
  16. 2020Complex Query Answering with Neural Link Predictorscited by 98
  17. 2023ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agentcited by 78
  18. 2023FactCHD: Benchmarking Fact-Conflicting Hallucination Detectioncited by 29
  19. 2023TEILP: Time Prediction over Knowledge Graphs via Logical Reasoningcited by 23
  20. 2022Neural-Symbolic Entangled Framework for Complex Query Answeringcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 13; 2026: 6.

24 papers.

Sort:

By area:

  1. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learningcited by 152
  2. 2024KAG: Boosting LLMs in Professional Domains via Knowledge Augmented Generationcited by 67
  3. 2024Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuningcited by 40
  4. 2025Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMscited by 40
  5. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  6. 2025Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and Opportunitiescited by 30
  7. 2024Knowledge-Aware Query Expansion with Large Language Models for Textual and Relational Retrievalcited by 24
  8. 2025Test-Time Adaptation for LLM Agents via Environment Interactioncited by 17
  9. 2025Ontology-Guided Reverse Thinking Makes Large Language Models Stronger on Knowledge Graph Question Answeringcited by 15
  10. 2024RuleRAG: Rule-Guided Retrieval-Augmented Generation with Language Models for Question Answeringcited by 13
  11. 2025Credible Plan-Driven RAG Method for Multi-Hop Question Answeringcited by 12
  12. 2025Large Language Models for Knowledge Graph Embedding: A Surveycited by 3
12 more
  1. 2025From Symbolic to Neural and Back: Exploring Knowledge Graph-Large Language Model Synergiescited by 3
  2. 2026Only Ask What You Don't Know: Grounded Delta Planning for Efficient Multi-step RAGcited by 3
  3. 2025GLTW: Joint Improved Graph Transformer and LLM via Three-Word Language for Knowledge Graph Completioncited by 2
  4. 2025Enhancing LLM Tool Use with High-quality Instruction Data from Knowledge Graphcited by 2
  5. 2024A Survey of Query Optimization in Large Language Modelscited by 1
  6. 2025KunLunBaizeRAG: Reinforcement Learning Driven Inference Performance Leap for Large Language Modelscited by 0
  7. 2025Avoiding Knowledge Edit Skipping in Multi-hop Question Answering with Guided Decompositioncited by 0
  8. 2026VulReaD: Knowledge-Graph-guided Software Vulnerability Reasoning and Detectioncited by 0
  9. 2026WebClipper: Efficient Evolution of Web Agents with Graph-based Trajectory Pruningcited by 0
  10. 2026RealRoute: Dynamic Query Routing System via Retrieve-then-Verify Paradigmcited by 0
  11. 2026LLMs+Graphs: Toward Graph-Native, Synergistic AI Systemscited by 0
  12. 2026KARMA: Knowledge graph-based Automated Reasoning Materialization and Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023KnowledgeNavigator: Leveraging Large Language Models for Enhanced Reasoning over Knowledge Graph
  2. 2025Plan Then Retrieve: Reinforcement Learning-Guided Complex Reasoning over Knowledge Graphs
  3. 2024KARPA: A Training-free Method of Adapting Knowledge Graph as References for Large Language Model's Reasoning Path Aggregation
  4. 2025Exemplar-Guided Planing: Enhanced LLM Agent for KGQA
  5. 2025Plan of Knowledge: Retrieval-Augmented Large Language Models for Temporal Knowledge Graph Question Answering
  6. 2026A Human-in-the-Loop, LLM-Centered Architecture for Knowledge-Graph Question Answering
  7. 2025Self-Reflective Planning with Knowledge Graphs: Enhancing LLM Reasoning Reliability for Question Answering
  8. 2025RJE: A Retrieval-Judgment-Exploration Framework for Efficient Knowledge Graph Question Answering with LLMs