# Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories

arXiv: https://arxiv.org/abs/2506.18824 · HTML: https://mlsanity.com/paper/2506.18824/

## Key facts

- Cited by 81 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 31 of its 64 references that are in the map.
- First posted on arXiv 23 Jun 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 30 of 31)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://mlsanity.com/paper/2305.11738/) (2023)
- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [Demystifying Long Chain-of-Thought Reasoning in LLMs](https://mlsanity.com/paper/2502.03373/) (2025)
- [SWE-smith: Scaling Data for Software Engineering Agents](https://mlsanity.com/paper/2504.21798/) (2025)
- [Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents](https://mlsanity.com/paper/2403.02502/) (2024)
- [MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution](https://mlsanity.com/paper/2403.17927/) (2024)
- [SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents](https://mlsanity.com/paper/2406.12952/) (2024)
- [SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement](https://mlsanity.com/paper/2410.20285/) (2024)
- [Can Large Language Models Really Improve by Self-critiquing Their Own Plans?](https://mlsanity.com/paper/2310.08118/) (2023)
- [SWE-Bench+: Enhanced Coding Benchmark for LLMs](https://mlsanity.com/paper/2410.06992/) (2024)
- [From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future](https://mlsanity.com/paper/2408.02479/) (2024)
- [SpecRover: Code Intent Extraction via LLMs](https://mlsanity.com/paper/2408.02232/) (2024)
- [MarsCode Agent: AI-native Automated Bug Fixing](https://mlsanity.com/paper/2409.00899/) (2024)
- [Interactive Debugging and Steering of Multi-Agent AI Systems](https://mlsanity.com/paper/2503.02068/) (2025)
- [Agents in Software Engineering: Survey, Landscape, and Vision](https://mlsanity.com/paper/2409.09030/) (2024)
- [AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectories](https://mlsanity.com/paper/2410.07706/) (2024)
- [Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent System](https://mlsanity.com/paper/2410.08115/) (2024)
- [Repo2Run: Automated Building Executable Environment for Code Repository at Scale](https://mlsanity.com/paper/2502.13681/) (2025)
- [EnvBench: A Benchmark for Automated Environment Setup](https://mlsanity.com/paper/2503.14443/) (2025)
- [UniDebugger: Hierarchical Multi-Agent Framework for Unified Software Debugging](https://mlsanity.com/paper/2404.17153/) (2024)
- [Evaluating Agent-based Program Repair at Google](https://mlsanity.com/paper/2501.07531/) (2025)
- [TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved?](https://mlsanity.com/paper/2412.02883/) (2024)
- [From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning](https://mlsanity.com/paper/2411.03817/) (2024)
- [Agentic Bug Reproduction for Effective Automated Program Repair at Google](https://mlsanity.com/paper/2502.01821/) (2025)
- [Beyond pip install: Evaluating LLM Agents for the Automated Installation of Python Projects](https://mlsanity.com/paper/2412.06294/) (2024)
- [Agentic AI Software Engineers: Programming with Trust](https://mlsanity.com/paper/2502.13767/) (2025)
- [Issue2Test: Generating Reproducing Test Cases from Issue Reports](https://mlsanity.com/paper/2503.16320/) (2025)
- [De-Hallucinator: Mitigating LLM Hallucinations in Code Generation Tasks via Iterative Grounding](https://mlsanity.com/paper/2401.01701/) (2024)
- [CigaR: Cost-efficient Program Repair with LLMs](https://mlsanity.com/paper/2402.06598/) (2024)

## Built on by (most cited first, 30 of 81)

- [SWE-chat: Coding Agent Interactions From Real Users in the Wild](https://mlsanity.com/paper/2604.20779/) (2026)
- [Understanding Code Agent Behaviour: An Empirical Study of Success and Failure Trajectories](https://mlsanity.com/paper/2511.00197/) (2025)
- [Agentic Much? Adoption of Coding Agents on GitHub](https://mlsanity.com/paper/2601.18341/) (2026)
- [From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws](https://mlsanity.com/paper/2606.06324/) (2026)
- [ContextBench: A Benchmark for Context Retrieval in Coding Agents](https://mlsanity.com/paper/2602.05892/) (2026)
- [Process-Centric Analysis of Agentic Software Systems](https://mlsanity.com/paper/2512.02393/) (2025)
- [From Flat Logs to Causal Graphs: Hierarchical Failure Attribution for LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2602.23701/) (2026)
- [Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architectures](https://mlsanity.com/paper/2604.03515/) (2026)
- [CodeTracer: Towards Traceable Agent States](https://mlsanity.com/paper/2604.11641/) (2026)
- [Wink: Recovering from Misbehaviors in Coding Agents](https://mlsanity.com/paper/2602.17037/) (2026)
- [AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation](https://mlsanity.com/paper/2605.12925/) (2026)
- [ARGUS: Defending LLM Agents Against Context-Aware Prompt Injection](https://mlsanity.com/paper/2605.03378/) (2026)
- [Failure as a Process: An Anatomy of CLI Coding Agent Trajectories](https://mlsanity.com/paper/2607.09510/) (2026)
- [FALAT: Tracing Failures in LLM Agent Trajectories via Dependency-Guided Search](https://mlsanity.com/paper/2606.00765/) (2026)
- [Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure](https://mlsanity.com/paper/2604.02547/) (2026)
- [TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems](https://mlsanity.com/paper/2605.26563/) (2026)
- [AgentStepper: Interactive Debugging of Software Development Agents](https://mlsanity.com/paper/2602.06593/) (2026)
- [Agentic Software Issue Resolution with Large Language Models: A Survey](https://mlsanity.com/paper/2512.22256/) (2025)
- [When Agents Fail: A Comprehensive Study of Bugs in LLM Agents with Automated Labeling](https://mlsanity.com/paper/2601.15232/) (2026)
- [Promises, Perils, and (Timely) Heuristics for Mining Coding Agent Activity](https://mlsanity.com/paper/2601.18345/) (2026)
- [Dynamic Cogeneration of Bug Reproduction Test in Agentic Program Repair](https://mlsanity.com/paper/2601.19066/) (2026)
- [Are Coding Agents Generating Over-Mocked Tests? An Empirical Study](https://mlsanity.com/paper/2602.00409/) (2026)
- [MASPrism: Lightweight Failure Attribution for Multi-Agent Systems Using Prefill-Stage Signals](https://mlsanity.com/paper/2605.07509/) (2026)
- [Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agents](https://mlsanity.com/paper/2606.05391/) (2026)
- [Coherence Collapse: Diagnosing Why Code Agents Fail After Reaching the Right Code](https://mlsanity.com/paper/2603.24631/) (2026)
- [LLM-as-Code: Agentic Programming for Agent Harness](https://mlsanity.com/paper/2606.15874/) (2026)
- [What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents](https://mlsanity.com/paper/2607.06184/) (2026)
- [Fail-Fast, Restart-Smart: Early Failure Prediction and Restart for SWE Agentic Tasks](https://mlsanity.com/paper/2608.03222/) (2026)
- [CodeScout: Contextual Problem Statement Enhancement for Software Agents](https://mlsanity.com/paper/2603.05744/) (2026)
- [Debugging the Debuggers: Failure-Anchored Structured Recovery for Software Engineering Agents](https://mlsanity.com/paper/2605.08717/) (2026)

## Abstract (arXiv)

Large Language Model (LLM)-based agents are increasingly employed to automate complex software engineering tasks, such as program repair and issue resolution. These agents operate by autonomously generating natural language thoughts, invoking external tools, and iteratively refining their solutions. Despite their widespread adoption, the internal decision-making processes of these agents remain largely unexplored, limiting our understanding of their operational dynamics and failure modes. In this paper, we present a large-scale empirical study of the thought-action-result trajectories of three state-of-the-art LLM-based agents: RepairAgent, AutoCodeRover, and OpenHands. We unify their interaction logs into a common format, capturing 120 trajectories and 2,822 LLM interactions focused on program repair and issue resolution. Our study combines quantitative analyses of structural properties, action patterns, and token usage with qualitative assessments of reasoning coherence and feedback integration. We identify key trajectory characteristics, such as iteration counts and token consumption, recurring action sequences, and the semantic coherence of thoughts, actions, and their results. Our findings reveal behavioral motifs and anti-patterns that distinguish successful from failed executions, providing actionable insights for improving agent design, including prompting strategies, failure diagnosis, and anti-pattern detection. We release our dataset and annotation framework to support further research on transparent and robust autonomous software engineering agents.