Ranked by the Matcher: A Reproducibility Audit of Knowledge Graph Extraction from Threat Reports
Not on the map. Why?
Abstract (arXiv)
Security teams and researchers choose knowledge-graph extraction tooling for threat reports on the strength of published triple-F1 scores, yet those scores depend on how predicted triples are matched to gold annotations. We could reimplement the stated matching rule for only five of twelve inspected systems. Re-scoring ten system outputs on shared documents under eight protocols reverses eleven of forty-five pairwise orderings; one fixed prediction set spans 0.16--0.70 F1. On an external, human-adjudicated set, no mechanical matcher---lexical, embedding, or entailment---agrees with the reviewers more than seven times in ten; an LLM judge agrees far more often. To separate component effects from matcher rewards, we build CTIForge, whose deterministic validation layer can vary while extraction is held byte-identical. Across seven tested deployment configurations, no hosted backbone loses precision under validation and every offline one does. Backbone, decoding, and prompting covary across those configurations; on the one backbone we could serve both ways, serving alone reproduces the split. It coincides with a roughly 2.8-fold increase in actions explicitly disputing entity type, consistent with hand-written rules encoding the conventions of the extractor against which they were developed. We release the pipeline, protocol suite, and per-triple audit records.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 59 references that are in the map. First posted on arXiv 1 Sep 2026.
Builds on
26 of 59 references.
References · 26
- 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
- 2018TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Timecited by 108
- 2020Dos and Don'ts of Machine Learning in Computer Securitycited by 102
- 2023Revisiting Relation Extraction in the era of Large Language Modelscited by 102
- 2024Extract, Define, Canonicalize: An LLM-based Framework for Knowledge Graph Constructioncited by 70
- 2024CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligencecited by 62
- 2022Revisiting DocRED -- Addressing the False Negative Problem in Relation Extractioncited by 57
- 2024SEvenLLM: Benchmarking, Eliciting, and Enhancing Abilities of Large Language Models in Cyber Threat Intelligencecited by 22
- 2021What are the attackers doing now? Automating cyber threat intelligence extraction from text on pace with the changing threat landscape: A surveycited by 14
- 2024Actionable Cyber Threat Intelligence using Knowledge Graphs and Large Language Modelscited by 14
- 2024CTINexus: Automatic Cyber Threat Intelligence Knowledge Graph Construction Using Large Language Modelscited by 14
- 2025Improving Automatic Evaluation of Large Language Models (LLMs) in Biomedical Relation Extraction via LLMs-as-the-Judgecited by 13
14 more
- 2024GenRES: Rethinking Evaluation for Generative Relation Extraction in the Era of Large Language Modelscited by 7
- 2024Evaluating Generative Language Models in Information Extraction as Subjective Question Correctioncited by 6
- 2024CRUcialG: Reconstruct Integrated Attack Scenario Graphs by Cyber Threat Intelligence Reportscited by 5
- 2024AttacKG+:Boosting Attack Knowledge Graph Construction with Large Language Modelscited by 4
- 2026Beyond Single Reports: Evaluating Automated ATT&CK Technique Extraction in Multi-Report Campaign Settingscited by 2
- 2026Schema-Agnostic Knowledge Graph Construction via Hybrid Ontology Discovery for Cyber Threat Intelligencecited by 2
- 2024R+R: Revisiting Static Feature-Based Android Malware Detection using Machine Learningcited by 1
- 2026Context-aware Entity-Relation Extraction for Threat Intelligence Knowledge Graphscited by 1
- 2026GRID: Graph Representation of Intelligence Data for Security Text Knowledge Graph Constructioncited by 1
- 2026CTI2TTP: Evidence-Grounded TTP Extraction from Cyber Threat Intelligence via Diverge-then-Converge Verificationcited by 1
- 2026Better Later Than Sooner: Neuro-Symbolic Knowledge Graph Construction via Ontology-grounded Post-extraction Correctioncited by 1
- 2026Evaluating Open-Source LLMs for Multi-Label ATT&CK Technique Classification on CTI Reportscited by 1
- 2026TACTIC-KG: Toward Small Agent Teams for Cyber Threat Intelligence Knowledge Graph Constructioncited by 1
- 2026TRACE-CTI: Auditable Post-Extraction Governance of TTP Claims with Knowledge Graphscited by 1
Built on by
Not cited yet.