# OrgForge-IT: A Verifiable Synthetic Benchmark for LLM-Based Insider Threat Detection

arXiv: https://arxiv.org/abs/2603.22499 · HTML: https://mlsanity.com/paper/2603.22499/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 15 references that are in the map.
- First posted on arXiv 23 Mar 2026.

## Builds on (most cited first, 9 of 9)

- [Malware Detection by Eating a Whole EXE](https://mlsanity.com/paper/1710.09435/) (2017)
- [A Survey of Network-based Intrusion Detection Data Sets](https://mlsanity.com/paper/1903.02460/) (2019)
- [CORTEX: Collaborative LLM Agents for High-Stakes Alert Triage](https://mlsanity.com/paper/2510.00311/) (2025)
- [CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning](https://mlsanity.com/paper/2509.20166/) (2025)
- [Benchmarking LLM-Assisted Blue Teaming via Standardized Threat Hunting](https://mlsanity.com/paper/2509.23571/) (2025)
- [Chimera: Harnessing Multi-Agent LLMs for Automatic Insider Threat Simulation](https://mlsanity.com/paper/2508.07745/) (2025)
- [LLMLogAnalyzer: A Clustering-Based Log Analysis Chatbot using Large Language Models](https://mlsanity.com/paper/2510.24031/) (2025)
- [User-Based Sequential Modeling with Transformer Encoders for Insider Threat Detection](https://mlsanity.com/paper/2506.23446/) (2025)
- [An Ethically Grounded LLM-Based Approach to Insider Threat Synthesis and Detection](https://mlsanity.com/paper/2509.06920/) (2025)

## Built on by (most cited first, 5 of 5)

- [GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation](https://mlsanity.com/paper/2606.22737/) (2026)
- [Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings](https://mlsanity.com/paper/2607.21962/) (2026)
- [HBEE: Human Behavioral Entropy Engine -- Pre-Registered Multi-Agent LLM Simulation of Peer-Suspicion-Based Detection Inversion](https://mlsanity.com/paper/2605.07472/) (2026)
- [MOLE: Detecting Insider Threats in AI Agents](https://mlsanity.com/paper/2609.06966/) (2026)
- [When Should Enterprise Policy Decisions Be Learned, and When Should They Be Reasoned?](https://mlsanity.com/paper/2610.17643/) (2026)

## Abstract (arXiv)

Synthetic insider threat benchmarks face a consistency problem: corpora generated without an external factual constraint cannot rule out cross-artifact contradictions. The CERT dataset -- the field's canonical benchmark -- is also static, lacks cross-surface correlation scenarios, and predates the LLM era. We present OrgForge-IT, a verifiable synthetic benchmark in which a deterministic simulation engine maintains ground truth and language models generate only surface prose, making cross-artifact consistency an architectural guarantee. The corpus spans 51 simulated days, 2,904 telemetry records at a 96.4% noise rate, and four detection scenarios designed to defeat single-surface and single-day triage strategies across three threat classes and eight injectable behaviors. A ten-model leaderboard reveals several findings: (1) triage and verdict accuracy dissociate - eight models achieve identical triage F1=0.80 yet split between verdict F1=1.0 and 0.80; (2) baseline false-positive rate is a necessary companion to verdict F1, with models at identical verdict accuracy differing by two orders of magnitude on triage noise; (3) victim attribution in the vishing scenario separates tiers - Tier A models exonerate the compromised account holder while Tier B models detect the attack but misclassify the victim; (4) rigid multi-signal thresholds structurally exclude single-surface negligent insiders, demonstrating the necessity of parallel, threat-class-specific triage pipelines; and (5) agentic software-engineering training acts as a force multiplier for multi-day temporal correlation, but only when paired with frontier-level parameter scale. Finally, prompt sensitivity analysis reveals that unstructured prompts induce vocabulary hallucination, motivating a two-track scoring framework separating prompt adherence from reasoning capability. OrgForge-IT is open source under the MIT license.