# Honeyval: A Comprehensive Evaluation Framework for LLM-powered HTTP Honeypots

arXiv: https://arxiv.org/abs/2605.29963 · HTML: https://mlsanity.com/paper/2605.29963/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 42 references that are in the map.
- First posted on arXiv 28 May 2026.

## Builds on (most cited first, 16 of 16)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [PentestGPT: An LLM-empowered Automatic Penetration Testing Tool](https://mlsanity.com/paper/2308.06782/) (2023)
- [CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale](https://mlsanity.com/paper/2506.02548/) (2025)
- [EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities](https://mlsanity.com/paper/2409.16165/) (2024)
- [PentestAgent: Incorporating LLM Agents to Automated Penetration Testing](https://mlsanity.com/paper/2411.05185/) (2024)
- [BaxBench: Can LLMs Generate Correct and Secure Backends?](https://mlsanity.com/paper/2502.11844/) (2025)
- [D-CIPHER: Dynamic Collaborative Intelligent Multi-Agent System with Planner and Heterogeneous Executors for Offensive Security](https://mlsanity.com/paper/2502.10931/) (2025)
- [LLM in the Shell: Generative Honeypots](https://mlsanity.com/paper/2309.00155/) (2023)
- [Hacking CTFs with Plain Agents](https://mlsanity.com/paper/2412.02776/) (2024)
- [LLM Honeypot: Leveraging Large Language Models as Advanced Interactive Honeypot Systems](https://mlsanity.com/paper/2409.08234/) (2024)
- [HoneyGPT: Breaking the Trilemma in Terminal Honeypots with Large Language Model](https://mlsanity.com/paper/2406.01882/) (2024)
- [AutoBaxBuilder: Bootstrapping Code Security Benchmarking](https://mlsanity.com/paper/2512.21132/) (2025)
- [LLMHoney: A Real-Time SSH Honeypot with Large Language Model-Driven Dynamic Response Generation](https://mlsanity.com/paper/2509.01463/) (2025)
- [VelLMes: A high-interaction AI-based deception framework](https://mlsanity.com/paper/2510.06975/) (2025)
- [GPT-5 at CTFs: Case Studies From Top-Tier Cybersecurity Events](https://mlsanity.com/paper/2511.04860/) (2025)

## Built on by (most cited first, 1 of 1)

- [HoneyRoute: Honeypot-Model Routing for Adversarial LLM Serving](https://mlsanity.com/paper/2609.08306/) (2026)

## Abstract (arXiv)

Honeypots are decoy systems mimicking real system components designed to defend against cyber attacks. Recently, LLMs increasingly serve as simulation backbones for honeypots. They enable defenders to construct high-interaction honeypots with low system security risks. However, LLM-powered honeypot development lacks a unified evaluation framework. Most evaluations consist of measuring response similarity on fixed commands, manual testing, or real-world deployment. These methods are often not scalable for development, reproducible across evaluations, representative of practical attacks, or adaptable to various attacker and honeypot configurations. In this work, we bridge this gap and propose Honeyval, a comprehensive evaluation framework for LLM-powered HTTP honeypots. We address the limitations of prior evaluations by grounding the honeypots in 16 backend applications, using AI hacking agents as attackers, employing two control tasks to monitor agent and honeypot capabilities across customizations, and defining clear and verifiable exploit goals for the attacker. Using Honeyval, we conduct an extensive evaluation of recent cost-efficient LLMs as HTTP honeypots. Our experiments highlight the promise of LLM-powered honeypots; they lead to substantially longer interactions with the attacker than rule-based baseline honeypots and are far less frequently detected even by frontier models, all while, on average, preserving a running cost advantage against agentic attackers. Further, we experiment with different counter-offensive honeypots configurations, and observe unique trade-offs, such as longer interactions at the cost of increased detection.