Frugal Knowledge Graph Construction with Local LLMs: A Zero-Shot Pipeline, Self-Consistency and Wisdom of Artificial Crowds
On the map: LLM reasoning → Knowledge- and tool-augmented reasoning
also in 1 more
Abstract (arXiv)
This paper presents an empirical study of a multi-model zero-shot pipeline for knowledge graph construction and exploitation, executed entirely through local inference on consumer-grade hardware. We propose a reproducible evaluation framework integrating two external benchmarks (DocRED, HotpotQA), WebQuestionsSP-style synthetic data, and the RAGAS evaluation framework in an automated pipeline. On 500 document-level relations, our system achieves an F1 of 0.70 ± 0.041 in zero-shot, compared to 0.80 for supervised DREEAM. Text-to-query achieves an accuracy of 0.80 ± 0.06 on 200 samples. Multi-hop reasoning achieves an Exact Match (EM) of 0.46±0.04 on 500 HotpotQA questions, with a RAGAS faithfulness of 0.96 ± 0.04 on 50 samples. Beyond the pipeline, we study diversity mechanisms for difficult multi-hop reasoning. On 181 questions unsolvable at zero temperature, self-consistency (k=5, T =0.7) recovers up to 23% EM with a single Mixture-of-Experts (MoE) model, but the cross-model oracle (3 architectures x 5 samples) reaches 46.4%. We highlight an agreement paradox: strong consensus among samples signals collective hallucination rather than a reliable answer, echoing the work of Moussa"id et al. on the wisdom of crowds. Extending to the full pipeline (500 questions), self-consistency (k=3) raises EM from 0.46 to 0.48 ± 0.04. A confidence-routing cascade mechanism (Phi-4 rightarrow GPT-OSS, k=5) achieves an EM of 0.55 ± 0.04, the best result obtained, with 45.4% of questions rerouted. Finally, we show that V3 prompt engineering applied to other models does not reproduce the gains observed with Gemma-4, confirming the specific prompt/model interaction. The entire system runs in ~5 h on a single RTX 3090, without any training, for an estimated carbon footprint of 0.09 kg CO2 eq.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 16 of its 30 references that are in the map. First posted on arXiv 13 Apr 2026.
Builds on
16 of 30 references.
References · 16
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2022Survey of Hallucination in Natural Language Generationcited by 2,075
- 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
- 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
- 2019Quantifying the Carbon Emissions of Machine Learningcited by 521
- 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 418
- 2019DocRED: A Large-Scale Document-Level Relation Extraction Datasetcited by 239
- 2023Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulnesscited by 130
- 2020Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Poolingcited by 115
- 2023Revisiting Relation Extraction in the era of Large Language Modelscited by 102
4 more
- 2022Don't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environmentscited by 57
- 2023DREEAM: Guiding Attention with Evidence for Improving Document-Level Relation Extractioncited by 37
- 2013Social Influence and the Collective Dynamics of Opinion Formationcited by 16
- 2023Document-Level In-Context Few-Shot Relation Extraction via Pre-Trained Language Modelscited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Inference Scaled GraphRAG: Improving Multi Hop Question Answering on Knowledge Graphs
- 2025KG-o1: Enhancing Multi-hop Question Answering in Large Language Models via Knowledge Graph Integration
- 2025Diagnosing and Addressing Pitfalls in KG-RAG Datasets: Toward More Reliable Benchmarking
- 2026LivingRAG: Augmenting Graph RAG with Experience
- 2024Efficient Parallel Multi-Hop Reasoning: A Scalable Approach for Knowledge Graph Analysis
- 2025StepChain GraphRAG: Reasoning Over Knowledge Graphs for Multi-Hop Question Answering
- 2024MINTQA: A Multi-Hop Question Answering Benchmark for Evaluating LLMs on New and Tail Knowledge
- 2026The Graph Language: How Knowledge Graphs Speak to Large Language Models