mlsanityData to 14 Oct 2026

Synergizing LLMs and Knowledge Graphs: A Novel Approach to Software Repository-Related Question Answering

arXiv 2412.03815 ↗First posted 5 Dec 2024cs.SE cs.AI cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Software repositories contain valuable information for understanding the development process. However, extracting insights from repository data is time-consuming and requires technical expertise. While software engineering chatbots support natural language interactions with repositories, chatbots struggle to understand questions beyond their trained intents and to accurately retrieve the relevant data. This study aims to improve the accuracy of LLM-based chatbots in answering repository-related questions by augmenting them with knowledge graphs. We use a two-step approach: constructing a knowledge graph from repository data, and synergizing the knowledge graph with an LLM to handle natural language questions and answers. We curated 150 questions of varying complexity and evaluated the approach on five popular open-source projects. Our initial results revealed the limitations of the approach, with most errors due to the reasoning ability of the LLM. We therefore applied few-shot chain-of-thought prompting, which improved accuracy to 84%. We also compared against baselines (MSRBot and GPT-4o-search-preview), and our approach performed significantly better. In a task-based user study with 20 participants, users completed more tasks correctly and in less time with our approach, and they reported that it was useful. Our findings demonstrate that LLMs and knowledge graphs are a viable solution for making repository data accessible.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 32 of its 83 references that are in the map. First posted on arXiv 5 Dec 2024.

Builds on

32 of 83 references.

References · 32

  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  2. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,453
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  4. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  5. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
  6. 2020A Survey on Knowledge Graphs: Representation, Acquisition and Applicationscited by 643
  7. 2020Knowledge Graphscited by 621
  8. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  9. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
  10. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  11. 2023The Impact of AI on Developer Productivity: Evidence from GitHub Copilotcited by 274
  12. 2024GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinementscited by 103
20 more
  1. 2023Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learningcited by 72
  2. 2023Making Large Language Models Better Reasoners with Alignmentcited by 63
  3. 2023How Effective Are Neural Networks for Fixing Security Vulnerabilitiescited by 52
  4. 2024Can LLM Graph Reasoning Generalize beyond Pattern Memorization?cited by 47
  5. 2023Using Large Language Models to Generate, Validate, and Apply User Intent Taxonomiescited by 34
  6. 2024Knowledge Graphs as Context Sources for LLM-Based Explanations of Learning Recommendationscited by 31
  7. 2024Knowledge Graphs, Large Language Models, and Hallucinations: An NLP Perspectivecited by 29
  8. 2018What's in a GitHub Star? Understanding Repository Starring Practices in a Social Coding Platformcited by 25
  9. 2024Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practicecited by 25
  10. 2024Bridging Design and Development with Automated Declarative UI Code Generationcited by 19
  11. 2024GraphOTTER: Evolving LLM-based Graph Reasoning for Complex Table Question Answeringcited by 18
  12. 2023Improving Large Language Models in Event Relation Logical Predictioncited by 17
  13. 2024LLM-ARC: Enhancing LLMs with an Automated Reasoning Criticcited by 16
  14. 2020A Comparison of Natural Language Understanding Platforms for Chatbots in Software Engineeringcited by 10
  15. 2023Prompt Engineering or Fine-Tuning: An Empirical Assessment of LLMs for Codecited by 9
  16. 2024Debugging with Open-Source Large Language Models: An Evaluationcited by 7
  17. 2021GitHub Discussions: An Exploratory Study of Early Adoptioncited by 5
  18. 2019MSRBot: Using Bots to Answer Questions from Software Repositoriescited by 4
  19. 2019Migrating to GraphQL: A Practical Assessmentcited by 3
  20. 2023Predicting the First Response Latency of Maintainers and Contributors in Pull Requestscited by 3

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Retrieval-Augmented Code Generation: A Survey with Focus on Repository-Level Approachescited by 23

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering
  2. 2026DeepRepoQA: Code Repository Question Answering with Deep Agent Exploration
  3. 2025Fusing Knowledge and Language: A Comparative Study of Knowledge Graph-Based Question Answering with LLMs
  4. 2025SWE-QA: Can Language Models Answer Repository-level Code Questions?
  5. 2025SimpleDevQA: Benchmarking Large Language Models on Development Knowledge QA
  6. 2023DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text
  7. 2025RJE: A Retrieval-Judgment-Exploration Framework for Efficient Knowledge Graph Question Answering with LLMs
  8. 2025Building Specialized Software-Assistant ChatBot with Graph-Based Retrieval-Augmented Generation