mlsanityData to 7 Oct 2026

ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks

arXiv 2502.05352 ↗First posted 7 Feb 2025cs.AI cs.DC cs.MAcited by 48

Not on the map. Why?

Cited by 30 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Realizing the vision of using AI agents to automate critical IT tasks depends on the ability to measure and understand effectiveness of proposed solutions. We introduce ITBench, a framework that offers a systematic methodology for benchmarking AI agents to address real-world IT automation tasks. Our initial release targets three key areas: Site Reliability Engineering (SRE), Compliance and Security Operations (CISO), and Financial Operations (FinOps). The design enables AI researchers to understand the challenges and opportunities of AI agents for IT automation with push-button workflows and interpretable metrics. ITBench includes an initial set of 94 real-world scenarios, which can be easily extended by community contributions. Our results show that agents powered by state-of-the-art models resolve only 13.8% of SRE scenarios, 25.2% of CISO scenarios, and 0% of FinOps scenarios. We expect ITBench to be a key enabler of AI-driven IT automation that is correct, safe, and fast.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Feb 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 34.

48 papers.

Sort:

By area:

  1. 2025Survey on Evaluation of LLM-based Agentscited by 190
  2. 2025Measuring Agents in Productioncited by 35
  3. 2026ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspacescited by 31
  4. 2025BED-LLM: Intelligent Information Gathering with LLMs and Bayesian Experimental Designcited by 21
  5. 2025AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenancecited by 15
  6. 2026Cloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud Systemscited by 13
  7. 2025Taming Uncertainty via Automation: Observing, Analyzing, and Optimizing Agentic AI Systemscited by 12
  8. 2025Incident Response Planning Using a Lightweight Large Language Model with Reduced Hallucinationcited by 10
  9. 2026DevOps-Gym: Benchmarking AI Agents in Software DevOps Cyclecited by 9
  10. 2026LLMs Corrupt Your Documents When You Delegatecited by 9
  11. 2025NetArena: Dynamic Benchmarks for AI Agents in Network Automationcited by 8
  12. 2026SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarioscited by 8
36 more
  1. 2025Agentic Services Computingcited by 6
  2. 2026CSLE: A Reinforcement Learning Platform for Autonomous Security Managementcited by 5
  3. 2025STRIDE: A Systematic Framework for Selecting AI Modalities -- Agentic AI, AI Assistants, or LLM Callscited by 4
  4. 2026Graph-Free Root Cause Analysiscited by 3
  5. 2026PHMForge: Evaluating LLM Agents on Industrial Prognostics through MCP-Native, Algorithm-Grounded Toolscited by 3
  6. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  7. 2025PRAXIS: Integrating Program Analysis with Observability for Root-Cause Analysiscited by 2
  8. 2026Large Language Models for Agentic NetOps and AIOps: Architectures, Evaluation, and Safetycited by 2
  9. 2026ORCA-bench: How Ready Are Language Model Agents for Oncall?cited by 2
  10. 2025Declarative Techniques for NL Queries over Heterogeneous Datacited by 1
  11. 2026AI+HW 2035: Shaping the Next Decadecited by 1
  12. 2026CUJBench: Benchmarking LLM-Agent on Cross-Modal Failure Diagnosis from Browser to Backendcited by 1
  13. 2026PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documentscited by 1
  14. 2026Runtime-Structured Task Decomposition for Agentic Coding Systemscited by 1
  15. 2026Pooled Leaderboards Hide System-Specific Winners: A Reporting-Protocol Audit of Offline Root-Cause Analysis Benchmarkscited by 1
  16. 2026A Multi-Dataset Benchmark for Evaluating LLM Agents in Microservice Failure Diagnosiscited by 1
  17. 2026OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modelingcited by 1
  18. 2026InfraBench: Evaluating Infrastructure Agents Across Layers, Lifecycle, and Riskcited by 1
  19. 2026StarHarness: Evolving Harnesses with Stratified Search for Enterprise Environmentscited by 1
  20. 2025Representing Prompting Patterns with PDL: Compliance Agent Case Studycited by 0
  21. 2025LLM-Powered Fully Automated Chaos Engineering: Towards Enabling Anyone to Build Resilient Software Systems at Low Costcited by 0
  22. 2026A Context Engineering Framework for Improving Enterprise AI Agents based on Digital-Twin MDPcited by 0
  23. 2026Agentic Performance at the Edge: Insights from Benchmarkingcited by 0
  24. 2026Auditable Graph-Guided Root Cause Analysis for Kubernetes Incidentscited by 0
  25. 2026Synthetic Scenario Generation for Evaluation of Industry 4.0 Agentscited by 0
  26. 2026Towards a Systems Foundation for Agentic Cloud Managementcited by 0
  27. 2026Beyond Component Testing: Validating Agentic AI Systemscited by 0
  28. 2026Terminal Agents: A Survey of AI Agents in Command-Line Environmentscited by 0
  29. 2026ASTRA - Agentic System for Ticket Resolution and Analysiscited by 0
  30. 2026A Hybrid Dependency-Aware Framework for Task Decomposition and Dynamic Agent Generation in Oracle-to-PostgreSQL Migrationcited by 0
  31. 2026FDE-Bench: Evaluating LLM Agents for Deployment Environment Configurationcited by 0
  32. 2026The Last Human Gate: Forward Deployed Engineering for Governance Automationcited by 0
  33. 2026Coding Agents Aren't Enough! Evaluating an Enterprise Security Brain for Agentic Cloud Investigationscited by 0
  34. 2026SRE-Marathon: A Continuous, Change-Driven Benchmark for Autonomous Site Reliability Agentscited by 0
  35. 2026LLMs are General Asynchronous Agentscited by 0
  36. 2026Incident-Arena: Getting agents to the last nine of reliabilitycited by 0