# FDE-Bench: Evaluating LLM Agents for Deployment Environment Configuration

arXiv: https://arxiv.org/abs/2609.27571 · HTML: https://mlsanity.com/paper/2609.27571/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 25 of its 32 references that are in the map.
- First posted on arXiv 23 Sep 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 25 of 25)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024)
- [MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation](https://mlsanity.com/paper/2310.03302/) (2023)
- [RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts](https://mlsanity.com/paper/2411.15114/) (2024)
- [ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks](https://mlsanity.com/paper/2502.05352/) (2025)
- [AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds](https://mlsanity.com/paper/2501.06706/) (2025)
- [Repo2Run: Automated Building Executable Environment for Code Repository at Scale](https://mlsanity.com/paper/2502.13681/) (2025)
- [EnvBench: A Benchmark for Automated Environment Setup](https://mlsanity.com/paper/2503.14443/) (2025)
- [Beyond pip install: Evaluating LLM Agents for the Automated Installation of Python Projects](https://mlsanity.com/paper/2412.06294/) (2024)
- [SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments](https://mlsanity.com/paper/2507.09063/) (2025)
- [DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle](https://mlsanity.com/paper/2601.20882/) (2026)
- [NetArena: Dynamic Benchmarks for AI Agents in Network Automation](https://mlsanity.com/paper/2506.03231/) (2025)
- [Multi-Docker-Eval: A `Shovel of the Gold Rush' Benchmark on Automatic Environment Building for Software Engineering](https://mlsanity.com/paper/2512.06915/) (2025)
- [DeployBench: Benchmarking LLM Agents for Research Artifact Deployment](https://mlsanity.com/paper/2606.05238/) (2026)
- [ResearchEnvBench: Benchmarking Agents on Environment Synthesis for Research Code Execution](https://mlsanity.com/paper/2603.06739/) (2026)
- [Multi-IaC-Eval: Benchmarking Cloud Infrastructure as Code Across Multiple Formats](https://mlsanity.com/paper/2509.05303/) (2025)
- [RAT: RunAnyThing via Fully Automated Environment Configuration](https://mlsanity.com/paper/2604.23190/) (2026)
- [Verifier-First Evaluation of Agentic LLMs for Infrastructure-as-Code Generation](https://mlsanity.com/paper/2607.20478/) (2026)

## Abstract (arXiv)

Deployment requires an agent to turn application code into a running system whose services connect, become ready, and remain observable. FDE-Bench evaluates this capability with 136 deployment-configuration tasks spanning Docker images, multi-service Compose stacks, and Kubernetes, in greenfield and diagnose-and-repair modes. Agents submit declarative artifacts that are collected, rebuilt, and redeployed in a pristine environment. Four gated binary check layers measure build, readiness, behavior, and conformance to the deployment specification, using programmatic checks without an LLM judge. A four-arm release gate requires a resolving reference solution and rejects tasks solved by do-nothing, specification-transcription, or generic-stub submissions. The released check annotations expose the link between 2,145 checks and their specifications, including seven documented gaps. Three additional adversarial strategies test shortcuts in the grading signals; none resolves any of the 135 tasks they cover, while a vacuous health probe passes readiness and exposes the need for downstream checks. On the 136-task evaluation grid, seven language models from four providers use the same four-tool scaffold and resolve 52.9-75.0 percent of tasks. The three zero-intelligence floors resolve none and reach a mean Deployment Score of at most 0.44. Readiness is the largest failure stage, accounting for 110 of 313 unresolved episodes. Mean resolution rate is 30.7 percentage points higher on the repair task group than on the disjoint greenfield group, with a positive gap for every model; ten tasks resist all seven. In a 25-task case study, one practicing engineer directing Claude-Sonnet-5 resolves 92 percent against 72 percent for the autonomous baseline. FDE-Bench links deployment success and failure to artifacts that can be inspected and replayed.