# ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks

arXiv: https://arxiv.org/abs/2502.05352 · HTML: https://mlsanity.com/paper/2502.05352/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Feb 2025.

## Built on by (most cited first, 30 of 51)

- [Survey on Evaluation of LLM-based Agents](https://mlsanity.com/paper/2503.16416/) (2025)
- [Measuring Agents in Production](https://mlsanity.com/paper/2512.04123/) (2025)
- [ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces](https://mlsanity.com/paper/2604.05172/) (2026)
- [BED-LLM: Intelligent Information Gathering with LLMs and Bayesian Experimental Design](https://mlsanity.com/paper/2508.21184/) (2025)
- [AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenance](https://mlsanity.com/paper/2506.03828/) (2025)
- [Cloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud Systems](https://mlsanity.com/paper/2603.00468/) (2026)
- [Taming Uncertainty via Automation: Observing, Analyzing, and Optimizing Agentic AI Systems](https://mlsanity.com/paper/2507.11277/) (2025)
- [DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle](https://mlsanity.com/paper/2601.20882/) (2026)
- [Incident Response Planning Using a Lightweight Large Language Model with Reduced Hallucination](https://mlsanity.com/paper/2508.05188/) (2025)
- [SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarios](https://mlsanity.com/paper/2605.07161/) (2026)
- [NetArena: Dynamic Benchmarks for AI Agents in Network Automation](https://mlsanity.com/paper/2506.03231/) (2025)
- [LLMs Corrupt Your Documents When You Delegate](https://mlsanity.com/paper/2604.15597/) (2026)
- [Agentic Services Computing](https://mlsanity.com/paper/2509.24380/) (2025)
- [CSLE: A Reinforcement Learning Platform for Autonomous Security Management](https://mlsanity.com/paper/2604.15590/) (2026)
- [STRIDE: A Systematic Framework for Selecting AI Modalities -- Agentic AI, AI Assistants, or LLM Calls](https://mlsanity.com/paper/2512.02228/) (2025)
- [Graph-Free Root Cause Analysis](https://mlsanity.com/paper/2601.21359/) (2026)
- [PHMForge: Evaluating LLM Agents on Industrial Prognostics through MCP-Native, Algorithm-Grounded Tools](https://mlsanity.com/paper/2604.01532/) (2026)
- [ORCA-bench: How Ready Are Language Model Agents for Oncall?](https://mlsanity.com/paper/2607.28545/) (2026)
- [Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents](https://mlsanity.com/paper/2510.17491/) (2025)
- [PRAXIS: Integrating Program Analysis with Observability for Root-Cause Analysis](https://mlsanity.com/paper/2512.22113/) (2025)
- [Large Language Models for Agentic NetOps and AIOps: Architectures, Evaluation, and Safety](https://mlsanity.com/paper/2605.12729/) (2026)
- [OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling](https://mlsanity.com/paper/2608.05141/) (2026)
- [Declarative Techniques for NL Queries over Heterogeneous Data](https://mlsanity.com/paper/2510.16470/) (2025)
- [AI+HW 2035: Shaping the Next Decade](https://mlsanity.com/paper/2603.05225/) (2026)
- [CUJBench: Benchmarking LLM-Agent on Cross-Modal Failure Diagnosis from Browser to Backend](https://mlsanity.com/paper/2604.23455/) (2026)
- [PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documents](https://mlsanity.com/paper/2605.10341/) (2026)
- [Runtime-Structured Task Decomposition for Agentic Coding Systems](https://mlsanity.com/paper/2605.15425/) (2026)
- [What a Pooled Leaderboard Does Not Say: Auditing the Scope and Provenance of Method Comparisons in Microservice RCA Benchmarks](https://mlsanity.com/paper/2606.29159/) (2026)
- [A Multi-Dataset Benchmark for Evaluating LLM Agents in Microservice Failure Diagnosis](https://mlsanity.com/paper/2606.29193/) (2026)
- [InfraBench: Evaluating Infrastructure Agents Across Layers, Lifecycle, and Risk](https://mlsanity.com/paper/2608.11234/) (2026)

## Abstract (arXiv)

Realizing the vision of using AI agents to automate critical IT tasks depends on the ability to measure and understand effectiveness of proposed solutions. We introduce ITBench, a framework that offers a systematic methodology for benchmarking AI agents to address real-world IT automation tasks. Our initial release targets three key areas: Site Reliability Engineering (SRE), Compliance and Security Operations (CISO), and Financial Operations (FinOps). The design enables AI researchers to understand the challenges and opportunities of AI agents for IT automation with push-button workflows and interpretable metrics. ITBench includes an initial set of 94 real-world scenarios, which can be easily extended by community contributions. Our results show that agents powered by state-of-the-art models resolve only 13.8% of SRE scenarios, 25.2% of CISO scenarios, and 0% of FinOps scenarios. We expect ITBench to be a key enabler of AI-driven IT automation that is correct, safe, and fast.