# Improving Factuality in LLMs via Inference-Time Knowledge Graph Construction

arXiv: https://arxiv.org/abs/2509.03540 · HTML: https://mlsanity.com/paper/2509.03540/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 43 references that are in the map.
- First posted on arXiv 31 Aug 2025.

## Builds on (most cited first, 22 of 22)

- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [The Web as a Knowledge-base for Answering Complex Questions](https://mlsanity.com/paper/1803.06643/) (2018)
- [Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought](https://mlsanity.com/paper/2210.01240/) (2022)
- [Can We Edit Factual Knowledge by In-Context Learning?](https://mlsanity.com/paper/2305.12740/) (2023)
- [Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph](https://mlsanity.com/paper/2307.07697/) (2023)
- [Retrieval-Augmented Generation with Graphs (GraphRAG)](https://mlsanity.com/paper/2501.00309/) (2024)
- [Large-scale Simple Question Answering with Memory Networks](https://mlsanity.com/paper/1506.02075/) (2015)
- [Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering](https://mlsanity.com/paper/2306.04136/) (2023)
- [Learning From Mistakes Makes LLM Better Reasoner](https://mlsanity.com/paper/2310.20689/) (2023)
- [Crawling the Internal Knowledge-Base of Language Models](https://mlsanity.com/paper/2301.12810/) (2023)
- [Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely](https://mlsanity.com/paper/2409.14924/) (2024)
- [Retrieve-Rewrite-Answer: A KG-to-Text Enhanced LLMs Framework for Knowledge Graph Question Answering](https://mlsanity.com/paper/2309.11206/) (2023)
- [Retrieval-Augmented Generation with Conflicting Evidence](https://mlsanity.com/paper/2504.13079/) (2025)
- [Generate-on-Graph: Treat LLM as both Agent and KG in Incomplete Knowledge Graph Question Answering](https://mlsanity.com/paper/2404.14741/) (2024)
- [Beyond Chain-of-Thought, Effective Graph-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2305.16582/) (2023)
- [ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answers](https://mlsanity.com/paper/2110.06884/) (2021)
- [OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs](https://mlsanity.com/paper/2408.11832/) (2024)
- [Language Models can be Logical Solvers](https://mlsanity.com/paper/2311.06158/) (2023)
- [Automated Construction of Theme-specific Knowledge Graphs](https://mlsanity.com/paper/2404.19146/) (2024)
- [ReaRAG: Knowledge-guided Reasoning Enhances Factuality of Large Reasoning Models with Iterative Retrieval Augmented Generation](https://mlsanity.com/paper/2503.21729/) (2025)

## Built on by (most cited first, 2 of 2)

- [MemWeaver: Weaving Hybrid Memories for Traceable Long-Horizon Agentic Reasoning](https://mlsanity.com/paper/2601.18204/) (2026)
- [DepthCharge: A Domain-Agnostic Framework for Measuring Depth-Dependent Knowledge in Large Language Models](https://mlsanity.com/paper/2603.23514/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) often struggle with producing factually consistent answers due to limitations in their parametric memory. Retrieval-Augmented Generation (RAG) paradigms mitigate this issue by incorporating external knowledge at inference time. However, such methods typically handle knowledge as unstructured text, which reduces retrieval accuracy, hinders compositional reasoning, and amplifies the influence of irrelevant information on the factual consistency of LLM outputs. To overcome these limitations, we propose a novel framework that dynamically constructs and expands knowledge graphs (KGs) during inference, integrating both internal knowledge extracted from LLMs and external knowledge retrieved from external sources. Our method begins by extracting a seed KG from the question via prompting, followed by iterative expansion using the LLM's internal knowledge. The KG is then selectively refined through external retrieval, enhancing factual coverage and correcting inaccuracies. We evaluate our approach on three diverse Factual QA benchmarks, demonstrating consistent gains in factual accuracy over baselines. Our findings reveal that inference-time KG construction is a promising direction for enhancing LLM factuality in a structured, interpretable, and scalable manner.