# Process-Level Trajectory Evaluation for Environment Configuration in Software Engineering Agents

arXiv: https://arxiv.org/abs/2510.25694 · HTML: https://mlsanity.com/paper/2510.25694/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 17 of its 22 references that are in the map.
- First posted on arXiv 29 Oct 2025.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 17 of 17)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems](https://mlsanity.com/paper/2306.03091/) (2023)
- [SWE-smith: Scaling Data for Software Engineering Agents](https://mlsanity.com/paper/2504.21798/) (2025)
- [CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges](https://mlsanity.com/paper/2401.07339/) (2024)
- [AutoCodeRover: Autonomous Program Improvement](https://mlsanity.com/paper/2404.05427/) (2024)
- [Large Language Models for Software Engineering: Survey and Open Problems](https://mlsanity.com/paper/2310.03533/) (2023)
- [LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead](https://mlsanity.com/paper/2404.04834/) (2024)
- [You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects](https://mlsanity.com/paper/2412.10133/) (2024)
- [EnvBench: A Benchmark for Automated Environment Setup](https://mlsanity.com/paper/2503.14443/) (2025)
- [Automated Benchmark Generation for Repository-Level Coding Tasks](https://mlsanity.com/paper/2503.07701/) (2025)
- [Beyond pip install: Evaluating LLM Agents for the Automated Installation of Python Projects](https://mlsanity.com/paper/2412.06294/) (2024)
- [FlaPy: Mining Flaky Python Tests at Scale](https://mlsanity.com/paper/2305.04793/) (2023)

## Built on by (most cited first, 18 of 18)

- [SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios](https://mlsanity.com/paper/2512.18470/) (2025)
- [SWE-Universe: Scale Real-World Verifiable Environments to Millions](https://mlsanity.com/paper/2602.02361/) (2026)
- [CodeTracer: Towards Traceable Agent States](https://mlsanity.com/paper/2604.11641/) (2026)
- [Youtu-LLM: Unlocking the Native Agentic Potential for Lightweight Large Language Models](https://mlsanity.com/paper/2512.24618/) (2025)
- [TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents](https://mlsanity.com/paper/2606.28480/) (2026)
- [AdmTree: Compressing Lengthy Context with Adaptive Semantic Trees](https://mlsanity.com/paper/2512.04550/) (2025)
- [EvoConfig: Self-Evolving Multi-Agent Systems for Efficient Autonomous Environment Configuration](https://mlsanity.com/paper/2601.16489/) (2026)
- [Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understanding](https://mlsanity.com/paper/2603.18472/) (2026)
- [DeployBench: Benchmarking LLM Agents for Research Artifact Deployment](https://mlsanity.com/paper/2606.05238/) (2026)
- [From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AI](https://mlsanity.com/paper/2606.14502/) (2026)
- [What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents](https://mlsanity.com/paper/2607.06184/) (2026)
- [TangramPuzzle: Evaluating Multimodal Large Language Models with Compositional Spatial Reasoning](https://mlsanity.com/paper/2601.16520/) (2026)
- [MEnvAgent: Scalable Polyglot Environment Construction for Verifiable Software Engineering](https://mlsanity.com/paper/2601.22859/) (2026)
- [Learning to Control LLM Agent Harnesses with Offline Reinforcement Learning](https://mlsanity.com/paper/2607.05458/) (2026)
- [TopoAgent: A Self-Evolving Topological Agent for Multimodal Scientific Reasoning](https://mlsanity.com/paper/2607.14658/) (2026)
- [Self-Evolving Coding Agents](https://mlsanity.com/paper/2608.03392/) (2026)
- [AblateCell: A Reproduce-then-Ablate Agent for Virtual Cell Repositories](https://mlsanity.com/paper/2604.19606/) (2026)
- [From Atomic to Agentic: Towards Interpretable Evaluation of LLMs' Agentic Mathematical Capabilities](https://mlsanity.com/paper/2608.26950/) (2026)

## Abstract (arXiv)

Large language model-based agents show promise for software engineering, but environment configuration remains a bottleneck due to heavy manual effort and scarce large-scale, high-quality datasets. Existing benchmarks assess only end-to-end build/test success, obscuring where and why agents succeed or fail. We introduce the Environment Configuration Diagnosis Benchmark, Enconda-bench, which provides process-level trajectory assessment of fine-grained agent capabilities during environment setup-planning, perception-driven error diagnosis, feedback-driven repair, and action to execute final environment configuration. Our task instances are automatically constructed by injecting realistic README errors and are validated in Docker for scalable, high-quality evaluation. Enconda-bench combines process-level analysis with end-to-end executability to enable capability assessments beyond aggregate success rates. Evaluations across state-of-the-art LLMs and agent frameworks show that while agents can localize errors, they struggle to translate feedback into effective corrections, limiting end-to-end performance. To our knowledge, Enconda-bench is the first framework to provide process-level internal capability assessment for environment configuration, offering actionable insights for improving software engineering agents.