# AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds

arXiv: https://arxiv.org/abs/2501.06706 · HTML: https://mlsanity.com/paper/2501.06706/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 17 of its 77 references that are in the map.
- First posted on arXiv 12 Jan 2025.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 17 of 17)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [Augmented Language Models: a Survey](https://mlsanity.com/paper/2302.07842/) (2023)
- [Holistic Evaluation of Text-To-Image Models](https://mlsanity.com/paper/2311.04287/) (2023)
- [TaskWeaver: A Code-First Agent Framework](https://mlsanity.com/paper/2311.17541/) (2023)
- [Automatic Root Cause Analysis via Large Language Models for Cloud Incidents](https://mlsanity.com/paper/2305.15778/) (2023)
- [Automated Root Causing of Cloud Incidents using In-Context Learning with GPT-4](https://mlsanity.com/paper/2401.13810/) (2024)
- [Robust Multimodal Failure Detection for Microservice Systems](https://mlsanity.com/paper/2305.18985/) (2023)
- [Observability and Chaos Engineering on System Calls for Containerized Applications in Docker](https://mlsanity.com/paper/1907.13039/) (2019)

## Built on by (most cited first, 30 of 40)

- [A Survey of AIOps in the Era of Large Language Models](https://mlsanity.com/paper/2507.12472/) (2025)
- [Cloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud Systems](https://mlsanity.com/paper/2603.00468/) (2026)
- [DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle](https://mlsanity.com/paper/2601.20882/) (2026)
- [SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarios](https://mlsanity.com/paper/2605.07161/) (2026)
- [Coding Agents Are Guessing: Measuring Action-Boundary Violations in Underspecified DevOps Instructions](https://mlsanity.com/paper/2607.02294/) (2026)
- [NetArena: Dynamic Benchmarks for AI Agents in Network Automation](https://mlsanity.com/paper/2506.03231/) (2025)
- [Beyond Task Completion: An Assessment Framework for Evaluating Agentic AI Systems](https://mlsanity.com/paper/2512.12791/) (2025)
- [Open Agent Specification (Agent Spec): A Unified Representation for AI Agents](https://mlsanity.com/paper/2510.04173/) (2025)
- [Automated Cloud Infrastructure-as-Code Reconciliation with AI Agents](https://mlsanity.com/paper/2510.20211/) (2025)
- [CSLE: A Reinforcement Learning Platform for Autonomous Security Management](https://mlsanity.com/paper/2604.15590/) (2026)
- [OpenRCA 2.0: From Outcome Labels to Causal Process Supervision](https://mlsanity.com/paper/2606.27154/) (2026)
- [From Logic Monopoly to Social Contract: Separation of Power and the Institutional Foundations for Autonomous Agent Economies](https://mlsanity.com/paper/2603.25100/) (2026)
- [AOI: Context-Aware Multi-Agent Operations via Dynamic Scheduling and Hierarchical Memory Compression](https://mlsanity.com/paper/2512.13956/) (2025)
- [PRAXIS: Integrating Program Analysis with Observability for Root-Cause Analysis](https://mlsanity.com/paper/2512.22113/) (2025)
- [Large Language Models for Agentic NetOps and AIOps: Architectures, Evaluation, and Safety](https://mlsanity.com/paper/2605.12729/) (2026)
- [A Feature Engineering Approach for Business Impact-Oriented Failure Detection in Distributed Instant Payment Systems](https://mlsanity.com/paper/2510.21710/) (2025)
- [AOI: Turning Failed Trajectories into Training Signals for Autonomous Cloud Diagnosis](https://mlsanity.com/paper/2603.03378/) (2026)
- [VIGIL: Towards Edge-Extended Agentic AI for Enterprise IT Support](https://mlsanity.com/paper/2603.16110/) (2026)
- [GSAR: Typed Grounding for Hallucination Detection and Recovery in Multi-Agent LLMs](https://mlsanity.com/paper/2604.23366/) (2026)
- [CUJBench: Benchmarking LLM-Agent on Cross-Modal Failure Diagnosis from Browser to Backend](https://mlsanity.com/paper/2604.23455/) (2026)
- [What a Pooled Leaderboard Does Not Say: Auditing the Scope and Provenance of Method Comparisons in Microservice RCA Benchmarks](https://mlsanity.com/paper/2606.29159/) (2026)
- [InfraBench: Evaluating Infrastructure Agents Across Layers, Lifecycle, and Risk](https://mlsanity.com/paper/2608.11234/) (2026)
- [From Version Conflicts to Decision Conflicts: Selective Revalidation for Long-Running AI Agents](https://mlsanity.com/paper/2609.08015/) (2026)
- [LLM-Powered Fully Automated Chaos Engineering: Towards Enabling Anyone to Build Resilient Software Systems at Low Cost](https://mlsanity.com/paper/2511.07865/) (2025)
- [Interactive Clarification for Cloud Infrastructure-as-Code Synthesis](https://mlsanity.com/paper/2604.02382/) (2026)
- [OpsLLM: Construction of Large Language Model for Software Operations with Multi-stage Learning](https://mlsanity.com/paper/2605.02906/) (2026)
- [Synthetic Scenario Generation for Evaluation of Industry 4.0 Agents](https://mlsanity.com/paper/2607.22563/) (2026)
- [Evaluating Agentic Code Repair Capabilities in Distributed Systems](https://mlsanity.com/paper/2608.14863/) (2026)
- [Agent-Native Telemetry: Verifiable State-Delta Evidence for Autonomous Operations](https://mlsanity.com/paper/2608.16178/) (2026)
- [Simthesizer: An Agent-Driven Simulation Framework for LLM Serving Systems](https://mlsanity.com/paper/2608.24650/) (2026)

## Abstract (arXiv)

AI for IT Operations (AIOps) aims to automate complex operational tasks, such as fault localization and root cause analysis, to reduce human workload and minimize customer impact. While traditional DevOps tools and AIOps algorithms often focus on addressing isolated operational tasks, recent advances in Large Language Models (LLMs) and AI agents are revolutionizing AIOps by enabling end-to-end and multitask automation. This paper envisions a future where AI agents autonomously manage operational tasks throughout the entire incident lifecycle, leading to self-healing cloud systems, a paradigm we term AgentOps. Realizing this vision requires a comprehensive framework to guide the design, development, and evaluation of these agents. To this end, we present AIOPSLAB, a framework that not only deploys microservice cloud environments, injects faults, generates workloads, and exports telemetry data but also orchestrates these components and provides interfaces for interacting with and evaluating agents. We discuss the key requirements for such a holistic framework and demonstrate how AIOPSLAB can facilitate the evaluation of next-generation AIOps agents. Through evaluations of state-of-the-art LLM agents within the benchmark created by AIOPSLAB, we provide insights into their capabilities and limitations in handling complex operational tasks in cloud environments.