# How to Interpret Agent Behavior

arXiv: https://arxiv.org/abs/2605.13625 · HTML: https://mlsanity.com/paper/2605.13625/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 63 references that are in the map.
- First posted on arXiv 13 May 2026.

## Builds on (most cited first, 30 of 45)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [Why Do Multi-Agent LLM Systems Fail?](https://mlsanity.com/paper/2503.13657/) (2025)
- [Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models](https://mlsanity.com/paper/2310.04406/) (2023)
- [AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents](https://mlsanity.com/paper/2401.13178/) (2024)
- [Measuring Progress on Scalable Oversight for Large Language Models](https://mlsanity.com/paper/2211.03540/) (2022)
- [AI Agents That Matter](https://mlsanity.com/paper/2407.01502/) (2024)
- [Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents](https://mlsanity.com/paper/2403.02502/) (2024)
- [Measuring AI Ability to Complete Long Software Tasks](https://mlsanity.com/paper/2503.14499/) (2025)
- [TRAIL: Trace Reasoning and Agentic Issue Localization](https://mlsanity.com/paper/2505.08638/) (2025)
- [HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scale](https://mlsanity.com/paper/2409.16299/) (2024)
- [Reducing Cost of LLM Agents with Trajectory Reduction](https://mlsanity.com/paper/2509.23586/) (2025)
- [CellAgent: An LLM-driven Multi-Agent Framework for Automated Single-cell Data Analysis](https://mlsanity.com/paper/2407.09811/) (2024)
- [CollabCoder: A Lower-barrier, Rigorous Workflow for Inductive Collaborative Qualitative Analysis with Large Language Models](https://mlsanity.com/paper/2304.07366/) (2023)
- [AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts](https://mlsanity.com/paper/2601.11044/) (2026)
- [REPRO-Bench: Can Agentic AI Systems Assess the Reproducibility of Social Science Research?](https://mlsanity.com/paper/2507.18901/) (2025)
- [OPEx: A Component-Wise Analysis of LLM-Centric Agents in Embodied Instruction Following](https://mlsanity.com/paper/2403.03017/) (2024)
- [Auto-TA: Towards Scalable Automated Thematic Analysis (TA) via Multi-Agent Large Language Models with Reinforcement Learning](https://mlsanity.com/paper/2506.23998/) (2025)
- [Position: Scaling LLM Agents Requires Asymptotic Analysis with LLM Primitives](https://mlsanity.com/paper/2502.04358/) (2025)

## Built on by (most cited first, 10 of 10)

- [Sidekick: Designing Communication for Effective Multitasking with Computer Use Agents](https://mlsanity.com/paper/2607.17527/) (2026)
- [Using Grounded Theory for Agent Behavior Analysis at Scale](https://mlsanity.com/paper/2608.30391/) (2026)
- [Rhythms of Work: Multi-Scale Interpretation of Human Behavioral Traces for Workplace Agents](https://mlsanity.com/paper/2609.04556/) (2026)
- [Your Agent Says Yes: Interpreting Adversarial Market Behavior Beyond Individual Transactions](https://mlsanity.com/paper/2609.07675/) (2026)
- [AgentPProf: Semantic Profiler for Long Horizon AI Agents](https://mlsanity.com/paper/2609.20301/) (2026)
- [On the Behavioral Traits of LLM Agents](https://mlsanity.com/paper/2609.32776/) (2026)
- [AgentHabit: Characterizing Distinct Behaviors of Agents on Everyday Tasks](https://mlsanity.com/paper/2609.32795/) (2026)
- [Does an Agent's History Tell You When Compaction Will Hurt? A Modest, Bounded Effect on the TRACE Paired-Replay Corpus](https://mlsanity.com/paper/2610.08722/) (2026)
- [Bookkeeping, Composition, or Unreachable Gold? Reading MemoryAgentBench's Conflict-Resolution Scores Against a Frozen Last-Write Resolver](https://mlsanity.com/paper/2610.09193/) (2026)
- [Accurate but Not Humble: Evaluating Epistemic Humility in LLM Agents under Knowledge Conflict](https://mlsanity.com/paper/2610.12360/) (2026)

## Abstract (arXiv)

Autonomous agents such as Claude Code and Codex now operate for hours or even days. Understanding their runtime behavior has become critical for downstream tasks such as diagnosing inefficiencies, fixing bugs, and ensuring better oversight. A primary way to gain this understanding is analyzing the reasoning trajectories and execution traces these agents generate. Yet such data remains in unstructured natural-language form, making it difficult for humans to interpret at scale. We introduce ACT*ONOMY (a combination of Action and Taxonomy), a taxonomy for describing and analyzing agent behavior at runtime. ACT*ONOMY has two components: (1) the taxonomy itself, developed through Grounded Theory and structured as a three-level hierarchy of 10 actions, 46 subactions, and 120 leaf categories; and (2) an open repository that hosts the living taxonomy, provides an automated analysis pipeline that applies it to agent trajectories analysis, and defines an extension protocol for customization and growth. Our experiments show that ACTONOMY can compare behavioral profiles across agents and characterize a single agent's behavior across diverse trajectories, surfacing patterns indicative of failure modes. By providing a shared vocabulary, ACT*ONOMY helps researchers, agent designers, and end users interpret agent behavior more consistently, enabling better oversight and control.