mlsanityData to 7 Oct 2026

Sola-Visibility-ISPM: Benchmarking Agentic AI for Identity Security Posture Management Visibility

arXiv 2601.07880 ↗First posted 11 Jan 2026cs.CR cs.AIcited by 3

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Identity Security Posture Management (ISPM) is a core challenge for modern enterprises operating across cloud and SaaS environments. Answering basic ISPM visibility questions, such as understanding identity inventory and configuration hygiene, requires interpreting complex identity data, motivating growing interest in agentic AI systems. Despite this interest, there is currently no standardized way to evaluate how well such systems perform ISPM visibility tasks on real enterprise data. We introduce the Sola Visibility ISPM Benchmark, the first benchmark designed to evaluate agentic AI systems on foundational ISPM visibility tasks using a live, production-grade identity environment spanning AWS, Okta, and Google Workspace. The benchmark focuses on identity inventory and hygiene questions and is accompanied by the Sola AI Agent, a tool-using agent that translates natural-language queries into executable data exploration steps and produces verifiable, evidence-backed answers. Across 77 benchmark questions, the agent achieves strong overall performance, with an expert accuracy of 0.84 and a strict success rate of 0.77. Performance is highest on AWS hygiene tasks, where expert accuracy reaches 0.94, while results on Google Workspace and Okta hygiene tasks are more moderate, yet competitive. Overall, this work provides a practical and reproducible benchmark for evaluating agentic AI systems in identity security and establishes a foundation for future ISPM benchmarks covering more advanced identity analysis and governance tasks.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 16 of its 34 references that are in the map. First posted on arXiv 11 Jan 2026.

Builds on

16 of 34 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation14 Jul

Other references · 15

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  4. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  5. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
  6. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 223
  7. 2024Evaluation of Retrieval-Augmented Generation: A Surveycited by 146
  8. 2023LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarkscited by 94
  9. 2024CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligencecited by 61
  10. 2024SQLBench: A Comprehensive Evaluation for Text-to-SQL Capabilities of Large Language Modelscited by 32
  11. 2024SEvenLLM: Benchmarking, Eliciting, and Enhancing Abilities of Large Language Models in Cyber Threat Intelligencecited by 20
  12. 2024SECURE: Benchmarking Large Language Models for Cybersecuritycited by 13
3 more
  1. 2025CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoningcited by 12
  2. 2025OrgAccess: A Benchmark for Role Based Access Control in Organization Scale LLMscited by 8
  3. 2025GenSIaC: Toward Security-Aware Infrastructure-as-Code Generation with Large Language Modelscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026AI Native Asset Intelligencecited by 1
  2. 2026Cross-Vendor Sola ISPM Benchmark: Evaluating Agentic AI for Federated Identity Security Reasoningcited by 1
  3. 2026Coding Agents Aren't Enough! Evaluating an Enterprise Security Brain for Agentic Cloud Investigationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Open Security Benchmark: Towards Autonomous Enterprise Cyber Defense
  2. 2025Identity Management for Agentic AI: The new frontier of authorization, authentication, and security for an AI agent world
  3. 2025A Novel Zero-Trust Identity Framework for Agentic AI: Decentralized Authentication and Fine-Grained Access Control
  4. 2026BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click
  5. 2025Towards a Standard, Enterprise-Relevant Agentic AI Benchmark: Lessons from 5.5 billion tokens' worth of agentic AI evaluations
  6. 2026SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response
  7. 2026CTI-REALM: Benchmark to Evaluate Agent Performance on Security Detection Rule Generation Capabilities
  8. 2025Privacy in Action: Towards Realistic Privacy Mitigation and Evaluation for LLM-Powered Agents