mlsanityData to 14 Oct 2026

KEO: Knowledge Extraction on OMIn via Knowledge Graphs and RAG for Safety-Critical Aviation Maintenance

arXiv 2510.05524 ↗First posted 7 Oct 2025cs.CL cs.IRcited by 1

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

We present Knowledge Extraction on OMIn (KEO), a domain-specific knowledge extraction and reasoning framework with large language models (LLMs) in safety-critical contexts. Using the Operations and Maintenance Intelligence (OMIn) dataset, we construct a QA benchmark spanning global sensemaking and actionable maintenance tasks. KEO builds a structured Knowledge Graph (KG) and integrates it into a retrieval-augmented generation (RAG) pipeline, enabling more coherent, dataset-wide reasoning than traditional text-chunk RAG. We evaluate locally deployable LLMs (Gemma-3, Phi-4, Mistral-Nemo) and employ stronger models (GPT-4o, Llama-3.3) as judges. Experiments show that KEO markedly improves global sensemaking by revealing patterns and system-level insights, while text-chunk RAG remains effective for fine-grained procedural tasks requiring localized retrieval. These findings underscore the promise of KG-augmented LLMs for secure, domain-specific QA and their potential in high-stakes reasoning. The code is available at https://github.com/JonathanKarr33/keo.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 59 references that are in the map. First posted on arXiv 7 Oct 2025.

Builds on

31 of 59 references.

References · 31

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  3. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,453
  4. 2024A Survey on LLM-as-a-Judgecited by 1,423
  5. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  6. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  7. 2020Knowledge Graphscited by 621
  8. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  9. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 331
  10. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 257
  11. 2023Knowledge Graph Prompting for Multi-Document Question Answeringcited by 152
  12. 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 144
19 more
  1. 2023Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answeringcited by 119
  2. 2020Entities as Experts: Sparse Memory Access with Entity Supervisioncited by 100
  3. 2024Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answeringcited by 96
  4. 2025MedRAG: Enhancing Retrieval-augmented Generation with Knowledge Graph-Elicited Reasoning for Healthcare Copilotcited by 83
  5. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 72
  6. 2025Knowledge Graph-Guided Retrieval Augmented Generationcited by 58
  7. 2023Knowledge Graph-Augmented Language Models for Knowledge-Grounded Dialogue Generationcited by 54
  8. 2021Mention Memory: incorporating textual knowledge into Transformers through entity mention attentioncited by 43
  9. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 39
  10. 2025LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judgecited by 37
  11. 2024OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domaincited by 29
  12. 2023On the Evolution of Knowledge Graphs: A Survey and Perspectivecited by 10
  13. 2023FABULA: Intelligence Report Generation Using Retrieval-Augmented Narrative Constructioncited by 10
  14. 2025TexGS-VolVis: Expressive Scene Editing for Volume Visualization via Textured Gaussian Splattingcited by 9
  15. 2020MaintNet: A Collaborative Open-Source Library for Predictive Maintenance Language Resourcescited by 8
  16. 2025An Evaluation-Centric Paradigm for Scientific Visualization Agentscited by 6
  17. 2024Leveraging Variation Theory in Counterfactual Data Augmentation for Optimized Active Learningcited by 2
  18. 2025Trusted Knowledge Extraction for Operations and Maintenance Intelligencecited by 2
  19. 2025CAMB: A comprehensive industrial LLM benchmark on civil aviation maintenancecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026ARID: A Deployable Edge AI System for Structured Information Extraction from Industrial Maintenance Work Orderscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Building Trust in the Skies: A Knowledge-Grounded LLM-based Framework for Aviation Safety
  2. 2025RAGuard: A Novel Approach for in-context Safe Retrieval Augmented Generation for LLMs
  3. 2025NOTAM-Evolve: A Knowledge-Guided Self-Evolving Optimization Framework with LLMs for NOTAM Interpretation
  4. 2025Exploratory Semantic Reliability Analysis of Wind Turbine Maintenance Logs using Large Language Models
  5. 2026Evidence-Driven Reasoning for Industrial Maintenance Using Heterogeneous Data
  6. 2025FailureSensorIQ: A Multi-Choice QA Dataset for Understanding Sensor Relationships and Failure Modes
  7. 2025OneEval: Benchmarking LLM Knowledge-intensive Reasoning over Diverse Knowledge Bases
  8. 2026Structure-Grounded Knowledge Retrieval via Code Dependencies for Multi-Step Data Reasoning