# Ranked by the Matcher: A Reproducibility Audit of Knowledge Graph Extraction from Threat Reports

arXiv: https://arxiv.org/abs/2609.01671 · HTML: https://mlsanity.com/paper/2609.01671/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 59 references that are in the map.
- First posted on arXiv 1 Sep 2026.

## Builds on (most cited first, 26 of 26)

- [Humans or LLMs as the Judge? A Study on Judgement Biases](https://mlsanity.com/paper/2402.10669/) (2024)
- [TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time](https://mlsanity.com/paper/1807.07838/) (2018)
- [Dos and Don'ts of Machine Learning in Computer Security](https://mlsanity.com/paper/2010.09470/) (2020)
- [Revisiting Relation Extraction in the era of Large Language Models](https://mlsanity.com/paper/2305.05003/) (2023)
- [Extract, Define, Canonicalize: An LLM-based Framework for Knowledge Graph Construction](https://mlsanity.com/paper/2404.03868/) (2024)
- [CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence](https://mlsanity.com/paper/2406.07599/) (2024)
- [Revisiting DocRED -- Addressing the False Negative Problem in Relation Extraction](https://mlsanity.com/paper/2205.12696/) (2022)
- [SEvenLLM: Benchmarking, Eliciting, and Enhancing Abilities of Large Language Models in Cyber Threat Intelligence](https://mlsanity.com/paper/2405.03446/) (2024)
- [What are the attackers doing now? Automating cyber threat intelligence extraction from text on pace with the changing threat landscape: A survey](https://mlsanity.com/paper/2109.06808/) (2021)
- [Actionable Cyber Threat Intelligence using Knowledge Graphs and Large Language Models](https://mlsanity.com/paper/2407.02528/) (2024)
- [CTINexus: Automatic Cyber Threat Intelligence Knowledge Graph Construction Using Large Language Models](https://mlsanity.com/paper/2410.21060/) (2024)
- [Improving Automatic Evaluation of Large Language Models (LLMs) in Biomedical Relation Extraction via LLMs-as-the-Judge](https://mlsanity.com/paper/2506.00777/) (2025)
- [GenRES: Rethinking Evaluation for Generative Relation Extraction in the Era of Large Language Models](https://mlsanity.com/paper/2402.10744/) (2024)
- [Evaluating Generative Language Models in Information Extraction as Subjective Question Correction](https://mlsanity.com/paper/2404.03532/) (2024)
- [CRUcialG: Reconstruct Integrated Attack Scenario Graphs by Cyber Threat Intelligence Reports](https://mlsanity.com/paper/2410.11209/) (2024)
- [AttacKG+:Boosting Attack Knowledge Graph Construction with Large Language Models](https://mlsanity.com/paper/2405.04753/) (2024)
- [Beyond Single Reports: Evaluating Automated ATT&CK Technique Extraction in Multi-Report Campaign Settings](https://mlsanity.com/paper/2604.07470/) (2026)
- [Schema-Agnostic Knowledge Graph Construction via Hybrid Ontology Discovery for Cyber Threat Intelligence](https://mlsanity.com/paper/2606.01208/) (2026)
- [R+R: Revisiting Static Feature-Based Android Malware Detection using Machine Learning](https://mlsanity.com/paper/2409.07397/) (2024)
- [Context-aware Entity-Relation Extraction for Threat Intelligence Knowledge Graphs](https://mlsanity.com/paper/2605.15904/) (2026)
- [GRID: Graph Representation of Intelligence Data for Security Text Knowledge Graph Construction](https://mlsanity.com/paper/2605.16714/) (2026)
- [CTI2TTP: Evidence-Grounded TTP Extraction from Cyber Threat Intelligence via Diverge-then-Converge Verification](https://mlsanity.com/paper/2605.25836/) (2026)
- [Better Later Than Sooner: Neuro-Symbolic Knowledge Graph Construction via Ontology-grounded Post-extraction Correction](https://mlsanity.com/paper/2605.29168/) (2026)
- [Evaluating Open-Source LLMs for Multi-Label ATT&CK Technique Classification on CTI Reports](https://mlsanity.com/paper/2606.18166/) (2026)
- [TACTIC-KG: Toward Small Agent Teams for Cyber Threat Intelligence Knowledge Graph Construction](https://mlsanity.com/paper/2607.05001/) (2026)
- [TRACE-CTI: Auditable Post-Extraction Governance of TTP Claims with Knowledge Graphs](https://mlsanity.com/paper/2607.24563/) (2026)

## Abstract (arXiv)

Security teams and researchers choose knowledge-graph extraction tooling for threat reports on the strength of published triple-F1 scores, yet those scores depend on how predicted triples are matched to gold annotations. We could reimplement the stated matching rule for only five of twelve inspected systems. Re-scoring ten system outputs on shared documents under eight protocols reverses eleven of forty-five pairwise orderings; one fixed prediction set spans 0.16--0.70 F1. On an external, human-adjudicated set, no mechanical matcher---lexical, embedding, or entailment---agrees with the reviewers more than seven times in ten; an LLM judge agrees far more often. To separate component effects from matcher rewards, we build CTIForge, whose deterministic validation layer can vary while extraction is held byte-identical. Across seven tested deployment configurations, no hosted backbone loses precision under validation and every offline one does. Backbone, decoding, and prompting covary across those configurations; on the one backbone we could serve both ways, serving alone reproduces the split. It coincides with a roughly 2.8-fold increase in actions explicitly disputing entity type, consistent with hand-written rules encoding the conventions of the extractor against which they were developed. We release the pipeline, protocol suite, and per-triple audit records.