mlsanityData to 7 Oct 2026

DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle

arXiv 2601.20882 ↗First posted 27 Jan 2026cs.SE cs.AI cs.CRcited by 9

Not on the map. Why?

Abstract (arXiv)

Even though demonstrating extraordinary capabilities in code generation and software issue resolving, AI agents' capabilities in the full software DevOps cycle are still unknown. Different from pure code generation, handling the DevOps cycle in real-world software, including developing, deploying, and managing, requires analyzing large-scale projects, understanding dynamic program behaviors, leveraging domain-specific tools, and making sequential decisions. However, existing benchmarks focus on isolated problems and lack environments and tool interfaces for DevOps. We introduce DevOps-Gym, the first end-to-end benchmark for evaluating AI agents across core DevOps workflows: build and configuration, monitoring, issue resolving, and test generation. DevOps-Gym includes 700+ real-world tasks collected from 30+ projects in Java and Go. We develop a semi-automated data collection mechanism with rigorous and non-trivial expert efforts in ensuring the task coverage and quality. Our evaluation of state-of-the-art models and agents reveals fundamental limitations: they struggle with issue resolving and test generation in Java and Go, and remain unable to handle new tasks such as monitoring and build and configuration. These results highlight the need for essential research in automating the full DevOps cycle with AI agents.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 35 of its 49 references that are in the map. First posted on arXiv 27 Jan 2026.

Builds on

35 of 49 references.

References · 35

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2020Extracting Training Data from Large Language Modelscited by 2,144
  5. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  6. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  7. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  8. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  9. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  10. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 400
  11. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  12. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
23 more
  1. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278
  2. 2024AutoCodeRover: Autonomous Program Improvementcited by 272
  3. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  4. 2023Beyond Memorization: Violating Privacy Via Inference with Large Language Modelscited by 167
  5. 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 165
  6. 2024SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agentscited by 129
  7. 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agentscited by 111
  8. 2024EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 71
  9. 2025ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Taskscited by 48
  10. 2024Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Studycited by 40
  11. 2024TESTEVAL: Benchmarking Large Language Models for Test Case Generationcited by 38
  12. 2025BaxBench: Can LLMs Generate Correct and Secure Backends?cited by 38
  13. 2025AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Cloudscited by 36
  14. 2025EnvBench: A Benchmark for Automated Environment Setupcited by 36
  15. 2025Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworkscited by 36
  16. 2025GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveragingcited by 25
  17. 2025PatchPilot: A Cost-Efficient Software Engineering Agent with Early Attempts on Formal Verificationcited by 22
  18. 2024TestBench: Evaluating Class-Level Test Case Generation Capability of Large Language Modelscited by 15
  19. 2025ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environmentscited by 13
  20. 2019BugSwarm: Mining and Continuously Growing a Dataset of Reproducible Failures and Fixescited by 12
  21. 2025Co-PatcheR: Collaborative Software Patching with Component(s)-specific Small Reasoning Modelscited by 12
  22. 2024A Survey of using Large Language Models for Generating Infrastructure as Codecited by 10
  23. 2025ACSE-Eval: Can LLMs threat model real-world cloud infrastructure?cited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspacescited by 31
  2. 2026TermiGen: High-Fidelity Environment and Robust Trajectory Synthesis for Terminal Agentscited by 24
  3. 2026Managing Procedural Memory in LLM Agents: Control, Adaptation, and Evaluationcited by 8
  4. 2026OpenSage: Self-programming Agent Generation Enginecited by 4
  5. 2026MDGYM: Benchmarking AI Agents on Molecular Simulationscited by 3
  6. 2026InfraBench: Evaluating Infrastructure Agents Across Layers, Lifecycle, and Riskcited by 1
  7. 2026FDE-Bench: Evaluating LLM Agents for Deployment Environment Configurationcited by 0
  8. 2026LLMs are General Asynchronous Agentscited by 0
  9. 2026Incident-Arena: Getting agents to the last nine of reliabilitycited by 0