# Knowledge Graph-Guided Multi-Agent Distillation for Reliable Industrial Question Answering with Datasets

arXiv: https://arxiv.org/abs/2510.06240 · HTML: https://mlsanity.com/paper/2510.06240/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 36 of its 49 references that are in the map.
- First posted on arXiv 3 Oct 2025.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 30 of 36)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate](https://mlsanity.com/paper/2305.19118/) (2023)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications](https://mlsanity.com/paper/2402.07927/) (2024)
- [Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs](https://mlsanity.com/paper/2311.17371/) (2023)
- [Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents](https://mlsanity.com/paper/2201.04723/) (2022)
- [Exploring Human-Like Translation Strategy with Large Language Models](https://mlsanity.com/paper/2305.04118/) (2023)
- [Exploring Large Language Models for Knowledge Graph Completion](https://mlsanity.com/paper/2308.13916/) (2023)
- [GeoLM: Empowering Language Models for Geospatially Grounded Language Understanding](https://mlsanity.com/paper/2310.14478/) (2023)
- [Large Language Model Guided Knowledge Distillation for Time Series Anomaly Detection](https://mlsanity.com/paper/2401.15123/) (2024)
- [Spatio-Temporal Urban Knowledge Graph Enabled Mobility Prediction](https://mlsanity.com/paper/2111.03465/) (2021)
- [Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment](https://mlsanity.com/paper/2501.03486/) (2025)
- [Large Language Model Meets Graph Neural Network in Knowledge Distillation](https://mlsanity.com/paper/2402.05894/) (2024)
- [TransGPT: Multi-modal Generative Pre-trained Transformer for Transportation](https://mlsanity.com/paper/2402.07233/) (2024)
- [Multistage Collaborative Knowledge Distillation from a Large Language Model for Semi-Supervised Sequence Generation](https://mlsanity.com/paper/2311.08640/) (2023)
- [Unleashing the Power of Imbalanced Modality Information for Multi-modal Knowledge Graph Completion](https://mlsanity.com/paper/2402.15444/) (2024)
- [Active Object Detection with Knowledge Aggregation and Distillation from Large Models](https://mlsanity.com/paper/2405.12509/) (2024)
- [REMAC: Self-Reflective and Self-Evolving Multi-Agent Collaboration for Long-Horizon Robot Manipulation](https://mlsanity.com/paper/2503.22122/) (2025)
- [Code Generation and Algorithmic Problem Solving Using Llama 3.1 405B](https://mlsanity.com/paper/2409.19027/) (2024)
- [Active Large Language Model-based Knowledge Distillation for Session-based Recommendation](https://mlsanity.com/paper/2502.15685/) (2024)
- [DistillSeq: A Framework for Safety Alignment Testing in Large Language Models using Knowledge Distillation](https://mlsanity.com/paper/2407.10106/) (2024)

## Built on by (most cited first, 1 of 1)

- [AgentArk: Distilling Multi-Agent Intelligence into a Single LLM Agent](https://mlsanity.com/paper/2602.03955/) (2026)

## Abstract (arXiv)

Industrial question-answering (QA) systems require higher safety and reliability than general-purpose dialogue models, as errors in high-risk scenarios such as equipment fault diagnosis can have severe consequences. Although multi-agent large language models enhance reasoning depth, they suffer from uncontrolled iterations and unverifiable outputs, and conventional distillation methods struggle to transfer collaborative reasoning capabilities to lightweight, deployable student models. To address these challenges, we propose Knowledge Graph-guided Multi-Agent System Distillation (KG-MASD). Our approach formulates distillation as a Markov Decision Process and incorporates a knowledge graph as a verifiable structured prior to enrich state representation and ensure convergence. By integrating collaborative reasoning with knowledge grounding, KG-MASD generates high-confidence instruction-tuning data and jointly distills reasoning depth and verifiability into compact student models suitable for edge deployment. Experiments on an industrial QA dataset show that KG-MASD improves accuracy by 2.4 per cent to 20.1 per cent over baselines and significantly enhances reliability, enabling trustworthy AI deployment in safety-critical industrial scenarios. Code and data are available at https://github.com/erwinmsmith/KG-MAD/.