# Benchmarking LLMs in an Embodied Environment for Blue Team Threat Hunting

arXiv: https://arxiv.org/abs/2505.11901 · HTML: https://mlsanity.com/paper/2505.11901/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 May 2025.

## Built on by (most cited first, 4 of 4)

- [Uncovering Vulnerabilities of LLM-Assisted Cyber Threat Intelligence](https://mlsanity.com/paper/2509.23573/) (2025)
- [From Trace to Line: LLM Agent for Real-World OSS Vulnerability Localization](https://mlsanity.com/paper/2510.02389/) (2025)
- [SIR-Bench: Evaluating Investigation Depth in Security Incident Response Agents](https://mlsanity.com/paper/2604.12040/) (2026)
- [Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents](https://mlsanity.com/paper/2607.15263/) (2026)

## Abstract (arXiv)

As cyber threats continue to grow in scale and sophistication, blue team defenders increasingly require advanced tools to proactively detect and mitigate risks. Large Language Models (LLMs) offer promising capabilities for enhancing threat analysis. However, their effectiveness in real-world blue team threat-hunting scenarios remains insufficiently explored. In this paper, we present CYBERTEAM, a benchmark designed to guide LLMs in blue teaming practice. CYBERTEAM constructs an embodied environment in two stages. First, it models realistic threat-hunting workflows by capturing the dependencies among analytical tasks from threat attribution to incident response. Next, each task is addressed through a set of embodied functions tailored to its specific analytical requirements. This transforms the overall threat-hunting process into a structured sequence of function-driven operations, where each node represents a discrete function and edges define the execution order. Guided by this framework, LLMs are directed to perform threat-hunting tasks through modular steps. Overall, CYBERTEAM integrates 30 tasks and 9 embodied functions, guiding LLMs through pipelined threat analysis. We evaluate leading LLMs and state-of-the-art cybersecurity agents, comparing CYBERTEAM's embodied function-calling against fundamental elicitation strategies. Our results offer valuable insights into the current capabilities and limitations of LLMs in threat hunting, laying the foundation for the practical adoption in real-world cybersecurity applications.