Coding Agents Aren't Enough! Evaluating an Enterprise Security Brain for Agentic Cloud Investigations
On the map: LLM agents and tool use → Agent threats, guardrails and governance
Abstract (arXiv)
Cloud-security investigation is dominated by population tasks: which identities can read a data store, how many resources fail a control, what is reachable from another account. These resolve against a complete inventory, not a named object, so a partial answer to one is not a partial result but a different one. Coding agents can now be given read-only cloud credentials and asked to investigate directly, which raises the question of what a purpose-built security context layer still contributes. We evaluate the Sola Security Brain, a security intelligence layer whose relational substrate is resolved offline and whose security logic is evaluated against it at query time, against two coding agents operating the same live AWS environment through a read-only CLI, Claude Code and OpenAI Codex, over 28 investigation tasks. All three are scored by a blinded, tier-weighted, grounding-gated relative recall over one joint claim pool, so the scores share a denominator. The Sola Security Brain reaches 0.549 +/- 0.012 coverage against 0.340 +/- 0.011 for Claude Code and 0.281 +/- 0.006 for Codex, with the ordering identical in every grading draw. It leads 24 of 28 tasks from the cheapest model tier, at 17.8x and 20.8x lower cost per task than Claude Code and Codex. Beyond the aggregate, we describe an answer-level pattern we term sample-and-generalise: both agents enumerate a fraction of a large population, assert an unhedged universal negative, and disclose the sample size only in answer metadata rather than in the answer. In one task Claude Code reported that no bucket policies exist after checking four bucket families, in a sweep that sampled 40 of roughly 5,000 buckets, in an account where 65 buckets carry a wildcard-principal read grant.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 37 of its 55 references that are in the map. First posted on arXiv 24 Sep 2026.
Builds on
37 of 55 references.
Closest ancestors · 8
Same direction, newest first
- 2026DiagChain: A Diagnostic Benchmark for Evaluating LLM Agents on Evidence-Grounded Attack Chain Reconstruction4 Aug
- 2026Open Security Benchmark: Towards Autonomous Enterprise Cyber Defense29 Jul
- 2026SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response29 Jul
- 2026Cross-Vendor Sola ISPM Benchmark: Evaluating Agentic AI for Federated Identity Security Reasoning1 Jun
- 2026Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps21 Apr
- 2026SIR-Bench: Evaluating Investigation Depth in Security Incident Response Agents13 Apr
- 2026Sola-Visibility-ISPM: Benchmarking Agentic AI for Identity Security Posture Management Visibility11 Jan
- 2025ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation14 Jul
Other references · 29
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
- 2023Large Language Models are not Fair Evaluatorscited by 1,050
- 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
- 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
- 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
- 2024Long-form factuality in large language modelscited by 146
- 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 129
- 2025WideSearch: Benchmarking Agentic Broad Info-Seekingcited by 66
- 2025ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Taskscited by 48
- 2025When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generationcited by 46
- 2026DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agentscited by 44
17 more
- 2025Budget-Aware Tool Use Enables Effective Agent Scalingcited by 42
- 2025The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Modelscited by 24
- 2026Cloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud Systemscited by 13
- 2026Over-Searching in Search-Augmented Large Language Modelscited by 10
- 2026When Is Enough Not Enough? Illusory Completion in Search Agentscited by 5
- 2026Total Recall QA: A Verifiable Evaluation Suite for Deep Research Agentscited by 5
- 2025Benchmarking LLM-Assisted Blue Teaming via Standardized Threat Huntingcited by 4
- 2026Tool Receipts, Not Zero-Knowledge Proofs: Practical Hallucination Detection for AI Agentscited by 4
- 2026Knowledge Graphs as the Missing Data Layer for LLM-Based Industrial Asset Operationscited by 3
- 2026Benchmarking and Exploring the Capabilities of LLMs for Attack Investigationscited by 2
- 2026SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integrationcited by 2
- 2026Quantifying Overclaiming Propensity in Frontier LLM Agentscited by 2
- 2023TAC: Hybrid IAM Privilege Escalation Detectioncited by 1
- 2026AI Native Asset Intelligencecited by 1
- 2026Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agentscited by 1
- 2026Does Runtime Topology Context Improve LLM-Generated Kubernetes Security Patches?cited by 1
- 2026When Absence Is Evidence: Evaluating Completeness-Sensitive Negative Reasoning in Large Language Modelscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Agentic Cloud Decoys: A Deception-Driven Framework for Autonomous Intrusion Investigation
- 2026Infrastructure for the Agentic Web: Gap Analysis and Architecture from the Agentverse Platform
- 2026The Balkanization of Execution-Security Research for AI Coding Agents: Isolation, Access Control, and Time-of-Check-to-Time-of-Use Vulnerabilities
- 2026When Agents Handle Secrets: A Survey of Confidential Computing for Agentic AI
- 2026Data Agents Under Attack: Vulnerabilities in LLM-Driven Analytical Systems
- 2026AGL-1: The Enterprise AI Governance Layer as a Control Plane for Trusted Enterprise Intelligence
- 2026Dive into Claude Code: The Design Space of Today's and Future AI Agent Systems
- 2026Towards a Systems Foundation for Agentic Cloud Management