# InfraBench: Evaluating Infrastructure Agents Across Layers, Lifecycle, and Risk

arXiv: https://arxiv.org/abs/2608.11234 · HTML: https://mlsanity.com/paper/2608.11234/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 43 references that are in the map.
- First posted on arXiv 31 Jul 2026.

## Builds on (most cited first, 17 of 17)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks](https://mlsanity.com/paper/2502.05352/) (2025)
- [Hyperledger Fabric: A Distributed Operating System for Permissioned Blockchains](https://mlsanity.com/paper/1801.10228/) (2018)
- [AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds](https://mlsanity.com/paper/2501.06706/) (2025)
- [Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models](https://mlsanity.com/paper/2301.03797/) (2023)
- [Building AI Agents for Autonomous Clouds: Challenges and Design Principles](https://mlsanity.com/paper/2407.12165/) (2024)
- [Assess and Summarize: Improve Outage Understanding with Large Language Models](https://mlsanity.com/paper/2305.18084/) (2023)
- [DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle](https://mlsanity.com/paper/2601.20882/) (2026)
- [SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarios](https://mlsanity.com/paper/2605.07161/) (2026)
- [Design and Implementation of ARA Wireless Living Lab for Rural Broadband and Applications](https://mlsanity.com/paper/2408.00913/) (2024)
- [Why Do My Blockchain Transactions Fail? A Study of Hyperledger Fabric (Extended version)*](https://mlsanity.com/paper/2103.04681/) (2021)
- [How To Optimize My Blockchain? A Multi-Level Recommendation Approach](https://mlsanity.com/paper/2301.04719/) (2023)
- [Revisiting Computational Storage for Data Integrity and Security](https://mlsanity.com/paper/2504.15293/) (2025)
- [On Fault Tolerance of Data Storage Systems: A Holistic Perspective](https://mlsanity.com/paper/2507.03849/) (2025)

## Built on by (most cited first, 1 of 1)

- [Incident-Arena: Getting agents to the last nine of reliability](https://mlsanity.com/paper/2610.00648/) (2026)

## Abstract (arXiv)

Managing modern computing infrastructure has become a steadily harder problem due to the ever-increasing complexity. Recent advances in AI agents create a timely opportunity to automate infrastructure management tasks, but it remains unclear how well such agents can handle real-world infrastructure complexity. We present InfraBench, a benchmark suite for evaluating AI agents on realistic infrastructure tasks across the full system stack and full operational lifecycle with fine-grained risk assessment. Experiments with 15 agent-model configurations show that even the strongest agent cannot secure a full score across all tasks. Mean effective scores range from roughly 40% to 88% (with per-configuration standard errors of 6-12 points), repeating every task three times reveals that top configurations still pass only a fraction of their attempts, and per-check scoring exposes a general failure pattern: agents may routinely satisfy short-term objectives while leaving non-durable changes, broken distributed invariants, unsafe side effects, and uncleaned state behind. INFRABENCH, including its live leaderboard, tasks, and evaluation harness, is publicly available at infraben.ch.