Understanding Agent-Reactive Bugs at the Model-Harness Boundary: An Empirical Study of LLM Agent Issue Reports

arXiv 2607.15684 ↗First posted 17 Jul 2026cs.SEcited by 3

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

LLM agents span command-line interfaces (e.g., Codex) and agent frameworks (e.g., LangChain), integrating backend LLMs with harness code that parses model outputs, controls agent loops, and manages context. Both the harness and LLM-generated responses jointly shape an agent's execution. This architecture gives rise to bugs that cannot be readily understood by inspecting either component alone, because some bugs occur only when a particular LLM response elicits an abnormal reaction from the agent. Prior empirical studies of agent bugs have largely attributed failures either to limited model capabilities or to harness-side defects, such as outdated APIs and configuration misalignment, without characterizing these AR bugs. We conduct the first empirical study focused on agent-reactive (AR) bugs. Through manual analysis of 255 bug reports from Codex, Gemini-CLI, LangChain, and CrewAI, we construct a two-axis taxonomy covering observable symptoms and the LLM behaviors that trigger them. Our findings show that many AR bugs manifest as silent errors without well-defined test oracles, which makes detection difficult. The stochasticity of LLM responses further complicates bug reproduction. We additionally examine fixes proposed by users and implemented by developers. This analysis exposes a mismatch: users frequently advocate harness-side guardrails, whereas developers may attribute the issue to the LLM or respond slowly to user-proposed fixes. These findings point to the need for mechanisms that help users and developers understand the root causes and resolutions of AR bugs. Overall, the study highlights challenges specific to LLM agents and motivates the design of test oracles, reproduction support, and fault-localization techniques for AR bugs.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 39 of its 55 references that are in the map. First posted on arXiv 17 Jul 2026.

Builds on

39 of 55 references.

Closest ancestors · 6

Same direction, newest first

Other references · 33

2020Language Models are Few-Shot Learnerscited by 32,419
2022Training language models to follow instructions with human feedbackcited by 15,971
2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
2021Evaluating Large Language Models Trained on Codecited by 8,320
2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601

21 more

2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
2023AgentBench: Evaluating LLMs as Agentscited by 1,267
2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
2023GAIA: a benchmark for General AI Assistantscited by 1,128
2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 341
2017DeepXplore: Automated Whitebox Testing of Deep Learning Systemscited by 311
2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 194
2023An Empirical Study of the Non-determinism of ChatGPT in Code Generationcited by 137
2019Taxonomy of Real Faults in Deep Learning Systemscited by 90
2019A Comprehensive Study on Deep Learning Bug Characteristicscited by 65
2024Reducing Tool Hallucination via Reliability Alignmentcited by 39
2021Silent Bugs in Deep Learning Frameworks: An Empirical Study of Keras and TensorFlowcited by 25
2025When LLMs Meet API Documentation: Can Retrieval Augmentation Aid Code Generation Just as It Helps Developers?cited by 13
2025Instruction-Following Evaluation in Function Calling for Large Language Modelscited by 1
2025Can Emulating Semantic Translation Help LLMs with Code Translation? A Study Based on Pseudocodecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

Similar, not cited

Close in wording, no citation either way