# Performance Evaluation of LLMs in Automated RDF Knowledge Graph Generation

arXiv: https://arxiv.org/abs/2603.29878 · HTML: https://mlsanity.com/paper/2603.29878/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 52 references that are in the map.
- First posted on arXiv 6 Feb 2026.

## Builds on (most cited first, 20 of 20)

- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [Retrieval-Augmented Generation with Graphs (GraphRAG)](https://mlsanity.com/paper/2501.00309/) (2024)
- [Generate-on-Graph: Treat LLM as both Agent and KG in Incomplete Knowledge Graph Question Answering](https://mlsanity.com/paper/2404.14741/) (2024)
- [Exploring Large Language Models for Knowledge Graph Completion](https://mlsanity.com/paper/2308.13916/) (2023)
- [From human experts to machines: An LLM supported approach to ontology and knowledge graph construction](https://mlsanity.com/paper/2403.08345/) (2024)
- [A Survey of Large Language Models on Generative Graph Analytics: Query, Learning, and Applications](https://mlsanity.com/paper/2404.14809/) (2024)
- [UrbanKGent: A Unified Large Language Model Agent Framework for Urban Knowledge Graph Construction](https://mlsanity.com/paper/2402.06861/) (2024)
- [Can LLMs be Good Graph Judge for Knowledge Graph Construction?](https://mlsanity.com/paper/2411.17388/) (2024)
- [NAT-NL2GQL: A Novel Multi-Agent Framework for Translating Natural Language to Graph Query Language](https://mlsanity.com/paper/2412.10434/) (2024)
- [Zero- and Few-Shots Knowledge Graph Triplet Extraction with Large Language Models](https://mlsanity.com/paper/2312.01954/) (2023)
- [GKG-LLM: A Unified Framework for Generalized Knowledge Graph Construction](https://mlsanity.com/paper/2503.11227/) (2025)
- [Leveraging Large Language Models for Semantic Query Processing in a Scholarly Knowledge Graph](https://mlsanity.com/paper/2405.15374/) (2024)
- [FIRESPARQL: A LLM-based Framework for SPARQL Query Generation over Scholarly Knowledge Graphs](https://mlsanity.com/paper/2508.10467/) (2025)
- [Traceable LLM-based validation of statements in knowledge graphs](https://mlsanity.com/paper/2409.07507/) (2024)
- [OntoLogX: Ontology-Guided Knowledge Graph Extraction from Cybersecurity Logs with Large Language Models](https://mlsanity.com/paper/2510.01409/) (2025)
- [Assessing LLMs Suitability for Knowledge Graph Completion](https://mlsanity.com/paper/2405.17249/) (2024)
- [Large Language Models for Knowledge Graph Embedding: A Survey](https://mlsanity.com/paper/2501.07766/) (2025)
- [SDLog: A Deep Learning Framework for Detecting Sensitive Information in Software Logs](https://mlsanity.com/paper/2505.14976/) (2025)
- [Evaluating improvements on using Large Language Models (LLMs) for property extraction in the Open Research Knowledge Graph (ORKG)](https://mlsanity.com/paper/2502.10768/) (2025)
- [Adaptive and Efficient Log Parsing as a Cloud Service](https://mlsanity.com/paper/2504.09113/) (2025)

## Abstract (arXiv)

Cloud systems generate large, heterogeneous log data containing critical infrastructure, application, and security information. Transforming these logs into RDF triples enables their integration into knowledge graphs, improving interpretability, root-cause analysis, and cross-service reasoning beyond what raw logs allow. Large Language Models (LLMs) offer a promising approach to automate RDF knowledge graph generation; however, their effectiveness on complex cloud logs remains largely unexplored. In this paper, we evaluate multiple LLM architectures and prompting strategies for automated RDF extraction using a controlled framework with two pipelines for systematically processing semi-structured log data. The extraction pipeline integrates multiple LLMs to identify relevant entities and relationships, automatically generating subject-predicate-object triples. These outputs are evaluated using a dedicated validation pipeline with both syntactic and semantic metrics to assess accuracy, completeness, and quality. Due to the lack of public ground-truth datasets, we created a reference Log-to-KG dataset from OpenStack logs using manual annotation and ontology-driven methods, enabling objective baseline. Our analysis shows that Few-Shot learning is the most effective strategy, with Llama achieving a 99.35% F1 score and 100% valid RDF output while Qwen, NuExtract, and Gemma also perform well under Few-Shot prompting, with Chain-of-Thought approaches maintaining similar accuracy. One-Shot prompting offers a lighter but effective alternative, while Zero-Shot and advanced strategies such as Tree-of-Thought, Self-Critique, and Generate-Multiple perform substantially worse. These results highlight the importance of contextual examples and prompt design for accurate RDF extraction and reveal model-specific limitations across LLM architectures.