# SCRIPTIOC-BENCH: A Benchmark for Recognizing Actionable Threat Intelligence from Script-Based Malware using LLMs

arXiv: https://arxiv.org/abs/2609.06149 · HTML: https://mlsanity.com/paper/2609.06149/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 67 references that are in the map.
- First posted on arXiv 5 Sep 2026.

## Builds on (most cited first, 15 of 15)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Large Language Models for Code Analysis: Do LLMs Really Do Their Job?](https://mlsanity.com/paper/2310.12357/) (2023)
- [Large Language Models Lack Understanding of Character Composition of Words](https://mlsanity.com/paper/2405.11357/) (2024)
- [Assessing LLMs in Malicious Code Deobfuscation of Real-world Malware Campaigns](https://mlsanity.com/paper/2404.19715/) (2024)
- [Time for aCTIon: Automated Analysis of Cyber Threat Intelligence in the Wild](https://mlsanity.com/paper/2307.10214/) (2023)
- [CTINexus: Automatic Cyber Threat Intelligence Knowledge Graph Construction Using Large Language Models](https://mlsanity.com/paper/2410.21060/) (2024)
- [On Benchmarking Code LLMs for Android Malware Analysis](https://mlsanity.com/paper/2504.00694/) (2025)
- [CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning](https://mlsanity.com/paper/2509.20166/) (2025)
- [JsDeObsBench: Measuring and Benchmarking LLMs for JavaScript Deobfuscation](https://mlsanity.com/paper/2506.20170/) (2025)
- [PowerPeeler: A Precise and General Dynamic Deobfuscation Method for PowerShell Scripts](https://mlsanity.com/paper/2406.04027/) (2024)
- [Is "Knowing It's Malicious Enough?" Evaluating LLMs for Fine-Grained Malware Behavior Auditing](https://mlsanity.com/paper/2509.14335/) (2025)
- [Optimizing Away JavaScript Obfuscation](https://mlsanity.com/paper/2009.09170/) (2020)
- [StringLLM: Understanding the String Processing Capability of Large Language Models](https://mlsanity.com/paper/2410.01208/) (2024)
- [From Obfuscated to Obvious: A Comprehensive JavaScript Deobfuscation Tool for Security Analysis](https://mlsanity.com/paper/2512.14070/) (2025)
- [PowerDrive: Accurate De-Obfuscation and Analysis of PowerShell Malware](https://mlsanity.com/paper/1904.10270/) (2019)

## Abstract (arXiv)

Script-based malware remains a prevalent attack technique. These scripts often contain indicators of compromise (IOCs) that provide actionable threat intelligence. However, statically recovering such indicators is challenging, as relevant values may be dispersed or transformed within code. Although large language models (LLMs) have shown promise in security analysis, their ability to recover IOCs from malicious scripts remains underexplored. We present SCRIPTIOC-BENCH, a benchmark for measuring static IOC extraction capability on real-world malicious scripts. The benchmark comprises 634 manually verified JavaScript, PowerShell, and VBScript malware samples covering four IOC types (URLs, domains, IP addresses, and filesystem artifacts). We further stratify ground-truth IOCs by recovery level, distinguishing directly exposed indicators from those requiring decoding or reconstruction. Using this benchmark, we evaluate a broad range of proprietary and open-weight LLMs and show that IOC recovery without execution remains challenging across model scales: the strongest model reaches only 65.4 F1. To characterize how recovery fails, we introduce a false-positive taxonomy and use it to compare the error profiles of the evaluated models. We further study two mitigations on a small open-weight model, deterministic string utilities and task-specific adaptation, finding that they provide complementary recovery gains, raise precision, and shift errors toward sample-grounded mismatches.