# Understanding Code Agent Behaviour: An Empirical Study of Success and Failure Trajectories

arXiv: https://arxiv.org/abs/2511.00197 · HTML: https://mlsanity.com/paper/2511.00197/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 31 Oct 2025.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 16 of 16)

- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024)
- [Understanding the planning of LLM agents: A survey](https://mlsanity.com/paper/2402.02716/) (2024)
- [RepairAgent: An Autonomous, LLM-Based Agent for Program Repair](https://mlsanity.com/paper/2403.17134/) (2024)
- [Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories](https://mlsanity.com/paper/2506.18824/) (2025)
- [SE-Agent: Self-Evolution Trajectory Optimization in Multi-Step Reasoning with LLM-Based Agents](https://mlsanity.com/paper/2508.02085/) (2025)
- [AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectories](https://mlsanity.com/paper/2410.07706/) (2024)
- [Neural Program Repair with Execution-based Backpropagation](https://mlsanity.com/paper/2105.04123/) (2021)
- [Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study](https://mlsanity.com/paper/2503.15223/) (2025)
- [SelfAPR: Self-supervised Program Repair with Test Execution Diagnostics](https://mlsanity.com/paper/2203.12755/) (2022)
- [ITER: Iterative Neural Repair for Multi-Location Patches](https://mlsanity.com/paper/2304.12015/) (2023)
- [Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study](https://mlsanity.com/paper/2506.08311/) (2025)
- [Prometheus: Towards Long-Horizon Codebase Navigation for Repository-Level Problem Solving](https://mlsanity.com/paper/2507.19942/) (2025)
- [When Fine-Tuning LLMs Meets Data Privacy: An Empirical Study of Federated Learning in LLM-Based Program Repair](https://mlsanity.com/paper/2412.01072/) (2024)
- [SeaView: Software Engineering Agent Visual Interface for Enhanced Workflow](https://mlsanity.com/paper/2504.08696/) (2025)
- [PreciseBugCollector: Extensible, Executable and Precise Bug-fix Collection](https://mlsanity.com/paper/2309.06229/) (2023)

## Built on by (most cited first, 30 of 33)

- [SWE-Pruner: Self-Adaptive Context Pruning for Coding Agents](https://mlsanity.com/paper/2601.16746/) (2026)
- [Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architectures](https://mlsanity.com/paper/2604.03515/) (2026)
- [Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?](https://mlsanity.com/paper/2607.09996/) (2026)
- [Wink: Recovering from Misbehaviors in Coding Agents](https://mlsanity.com/paper/2602.17037/) (2026)
- [AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation](https://mlsanity.com/paper/2605.12925/) (2026)
- [How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessions](https://mlsanity.com/paper/2605.29442/) (2026)
- [Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub](https://mlsanity.com/paper/2601.15195/) (2026)
- [Failure as a Process: An Anatomy of CLI Coding Agent Trajectories](https://mlsanity.com/paper/2607.09510/) (2026)
- [Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure](https://mlsanity.com/paper/2604.02547/) (2026)
- [Agentic Software Issue Resolution with Large Language Models: A Survey](https://mlsanity.com/paper/2512.22256/) (2025)
- [Coherence Collapse: Diagnosing Why Code Agents Fail After Reaching the Right Code](https://mlsanity.com/paper/2603.24631/) (2026)
- [What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents](https://mlsanity.com/paper/2607.06184/) (2026)
- [Fail-Fast, Restart-Smart: Early Failure Prediction and Restart for SWE Agentic Tasks](https://mlsanity.com/paper/2608.03222/) (2026)
- [Same Signal, Different Semantics: A Cross-Framework Behavioral Analysis of Software Engineering Agents](https://mlsanity.com/paper/2605.18332/) (2026)
- [Turning Interaction History into Execution State: A Runtime Layer for Long-Horizon Coding Agents](https://mlsanity.com/paper/2608.00808/) (2026)
- [The Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent Behavior](https://mlsanity.com/paper/2608.11386/) (2026)
- [EvoPolicyGym: Evaluating Autonomous Policy Evolution in Interactive Environments](https://mlsanity.com/paper/2607.02440/) (2026)
- [1D-Bench: A Benchmark for Iterative UI Code Generation with Visual Feedback in Real-World](https://mlsanity.com/paper/2602.18548/) (2026)
- [Measuring Cross-Task Behavioral Consistency in Language Model Agents](https://mlsanity.com/paper/2608.13598/) (2026)
- [The Working Set of a Coding Agent: Coherence Debt in Repository-Scale Tasks](https://mlsanity.com/paper/2608.16630/) (2026)
- [BootstrapAgent: Distilling Repository Setup into Reusable Agent Knowledge](https://mlsanity.com/paper/2605.15815/) (2026)
- [When and How Context Rot Appears in Coding Agents: A White-Box Study of Agent Skills in Code Auditing](https://mlsanity.com/paper/2607.17937/) (2026)
- [From Agent Behaviour to Agent-Friendly Documentation: An Empirical Study of How Coding Agents Discover, Read, and Write Technical Documentation](https://mlsanity.com/paper/2608.20195/) (2026)
- [openJiuwen: Beyond Static Harnesses for Long-Horizon Coding Agents](https://mlsanity.com/paper/2608.27969/) (2026)
- [Trajectory-Aware Benchmark Subset Selection for Cost-Efficient Software Engineering Agent Regression Testing](https://mlsanity.com/paper/2609.24928/) (2026)
- [Analyzing and Mitigating Cost-Inefficient Behaviors in Coding Agents](https://mlsanity.com/paper/2609.30725/) (2026)
- [DynBranch: Speculative Subgraph Reuse for Dynamic Agentic LLM Serving](https://mlsanity.com/paper/2609.31047/) (2026)
- [Learning When and How to Intervene: A Hindsight-Distilled Sentinel for Coding Agents](https://mlsanity.com/paper/2609.39957/) (2026)
- [Do Tool Calls Execute as Intended? Measuring and Repairing Intent-Execution Correspondence in LLM Agents](https://mlsanity.com/paper/2610.04375/) (2026)
- [Complex Agents, Shallow Tests: Demystifying and Enhancing Test Adequacy of Agent Harness in the Wild](https://mlsanity.com/paper/2610.04921/) (2026)

## Abstract (arXiv)

The increasing deployment of Large Language Model (LLM) agents for complex software engineering tasks has created a need to understand their problem-solving behaviours beyond simple success metrics. While these agents demonstrate impressive capabilities in automated issue resolution, their decision-making processes remain largely opaque. This paper presents an empirical study of agent trajectories, namely the execution traces capturing the steps agents take when attempting to resolve software issues. We analyse trajectories from three state-of-the-art code agents (OpenHands, SWE-agent, and Prometheus) on the SWE-Bench benchmark, examining both successful and failed attempts. Our investigation reveals several key insights into agent behaviour. First, we identify how distinct problem-solving strategies, such as defensive programming and context gathering, enable success in different scenarios. Second, we find that failed trajectories are consistently longer and exhibit higher variance than successful ones, with failure patterns differing significantly between agents. Third, our fault localisation analysis shows that while most trajectories correctly identify problematic files (72-81% even in failures), success depends more on achieving approximate rather than exact code modifications. These and other findings unveiled by our study, provide a foundation for understanding agent behaviour through trajectory analysis, contributing to the development of more robust and interpretable autonomous software engineering systems.