The Mechanism Matters: When Knowledge Graphs Help Reinforcement Learning
On the map: Deep reinforcement learning methods → RL training of language and agent models
Abstract (arXiv)
Knowledge graphs (KGs) are widely used to inject prior knowledge into reinforcement learning (RL), yet the literature is dominated by single-domain, positive-result method papers, so we lack a systematic account of when KG structure helps an agent, when it is neutral, and when it hurts. We conduct a controlled study that independently varies the RL task, the injection mechanism (state features, action masking, or potential-based reward shaping), and KG quality. Using a synthetic, fully controllable KG over MiniGrid environments, we report three findings. First, on compositional sparse-reward tasks structured KG guidance improves sample efficiency and solve reliability (70% to 97% of seeds), and a shuffle control that permutes the KG's edges while preserving their count collapses the benefit toward baseline (masking p=0.0001; shaping p=0.006), so the gain is structural rather than generic regularization. Second, KG value scales with the amount of task-relevant knowledge the graph contains. Third, and most consequential, safety depends on the mechanism: soft, optimality-preserving injection benefits from correct knowledge and harmlessly ignores incorrect knowledge, whereas hard masking is brittle, forbidding essential actions when the KG is incomplete or corrupted and making a wrong KG worse than none. A UMLS-derived clinical case study on sepsis management under offline RL is a careful null, underscoring that benefits require task structure the chosen mechanism can exploit. Our results give practitioners concrete guidance on how, and how much, to trust a KG when using it to guide RL.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 14 of its 24 references that are in the map. First posted on arXiv 21 Jul 2026.
Builds on
14 of 24 references.
References · 14
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
- 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
- 2017DeepPath: A Reinforcement Learning Method for Knowledge Graph Reasoningcited by 266
- 2019Batch Policy Learning under Constraintscited by 244
- 2017Go for a Walk and Arrive at the Answer: Reasoning Over Paths in Knowledge Bases using Reinforcement Learningcited by 161
- 2018Multi-Hop Knowledge Graph Reasoning with Reward Shapingcited by 102
- 2022Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexitycited by 62
- 2020Text-based RL Agents with Commonsense Knowledge: New Challenges, Environments and Baselinescited by 37
- 2021Learning Knowledge Graph-based World Models of Textual Environmentscited by 31
- 2019On Inductive Biases in Deep Reinforcement Learningcited by 24
- 2020Reward Propagation Using Graph Convolutional Networkscited by 15
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Enhancing Multi-Hop Knowledge Graph Reasoning through Reward Shaping Techniques
- 2022Value Memory Graph: A Graph-Structured World Model for Offline Reinforcement Learning
- 2023Foundations for Transfer in Reinforcement Learning: A Taxonomy of Knowledge Modalities
- 2026Knowledge Graphs are Implicit Reward Models: Path-Derived Signals Enable Compositional Reasoning
- 2025AutoGraph-R1: End-to-End Reinforcement Learning for Knowledge Graph Construction
- 2026Knowledge Graph-Driven Expert-Level Reasoning for Neuroscience
- 2024Reinforcement Learning Approach for Integrating Compressed Contexts into Knowledge Graphs
- 2026Knowledge- and Gradient-Guided Reinforcement Learning for Parametrized Action Markov Decision Processes