# Cost-Aware Hierarchical Multi-Agent Ransomware Detection and Family Attribution

arXiv: https://arxiv.org/abs/2609.04820 · HTML: https://mlsanity.com/paper/2609.04820/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 17 of its 37 references that are in the map.
- First posted on arXiv 4 Sep 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 17 of 17)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time](https://mlsanity.com/paper/1807.07838/) (2018)
- [Dos and Don'ts of Machine Learning in Computer Security](https://mlsanity.com/paper/2010.09470/) (2020)
- [Teams of LLM Agents can Exploit Zero-Day Vulnerabilities](https://mlsanity.com/paper/2406.01637/) (2024)
- [Large Language Models for Cyber Security: A Systematic Literature Review](https://mlsanity.com/paper/2405.04760/) (2024)
- [Exploring Adversarial Examples in Malware Detection](https://mlsanity.com/paper/1810.08280/) (2018)
- [Classification with Costly Features using Deep Reinforcement Learning](https://mlsanity.com/paper/1711.07364/) (2017)
- [Hierarchical Multi-agent Reinforcement Learning for Cyber Network Defense](https://mlsanity.com/paper/2410.17351/) (2024)
- [Adaptive Classification for Prediction Under a Budget](https://mlsanity.com/paper/1705.10194/) (2017)
- [A Survey on Ransomware: Evolution, Taxonomy, and Defense Solutions](https://mlsanity.com/paper/2102.06249/) (2021)
- [Prompt Engineering-assisted Malware Dynamic Analysis Using GPT-4](https://mlsanity.com/paper/2312.08317/) (2023)
- [SLIFER: Investigating Performance and Robustness of Malware Detection Pipelines](https://mlsanity.com/paper/2405.14478/) (2024)
- [Zero Day Ransomware Detection with Pulse: Function Classification with Transformer Models and Assembly Language](https://mlsanity.com/paper/2408.07862/) (2024)
- [Multimodal Multi-Agent Ransomware Analysis Using AutoGen](https://mlsanity.com/paper/2601.20346/) (2026)

## Abstract (arXiv)

Sandbox execution and memory forensics are among the most constrained resources in malware triage. Static analysis can scale to millions of files, whereas dynamic and memory analysis require minutes of analyst controlled infrastructure for each sample. Despite this difference, multimodal ransomware detectors often apply every modality to every sample, causing analysis cost and time to verdict to increase linearly with sample volume even when static evidence is already sufficient for a decision. We present a cost aware Hierarchical Multi-Agent System that formulates evidence acquisition as a budgeted sequential decision problem. Specialist agents generate schema validated risk signals for each modality, domain controllers aggregate these signals, and a Meta-Orchestrator begins with static evidence and escalates to dynamic and memory evidence only when confidence is insufficient or agents within a controller disagree. An optional, bounded, locally hosted large language model reviewer can adjust a verdict by at most one tier but cannot replace the deterministic pipeline. Each decision is recorded with a complete provenance trace. In multiple runs over 12439 samples from 16 ransomware families and benign samples, the deterministic HMAS achieves F1 0.93 and macro F1 0.97, resolving 57.95% of cases using static evidence alone, 35.83% after adding dynamic evidence, and only 6.21% through the full pipeline. The average internal analysis cost is 6.65 units, compared with 12 for exhaustive analysis, representing a 44.6% reduction. Standalone leave-one-family-out testing further shows that accuracy on families held out during tuning falls to 0.26 to 0.64 outside the Benign and high support classes. We report these results alongside a cost sensitivity analysis, a partial leave one component out ablation, and a full scale comparison with learned early and late fusion and cascade baselines.