Evaluation of Multi-Turn Consistency in LLM Agents: Survival Analysis and Failure-Rationale Taxonomy
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks
Abstract (arXiv)
Large language model (LLM) agents may perform well on isolated tasks yet drift into inconsistency over extended interaction. We evaluate temporal consistency in a controlled 20-step multi-agent setting inspired by delayed-gratification studies. At each step, an agent chooses between continuing to delay a reward or claiming it immediately (terminating the episode). Across a full-factorial manipulation of social visibility (private vs public), persona stressors, and deliberation policy, we run 84,540 trajectories spanning 8 model families. Treating the first reward-claim as a time-to-event outcome, we estimate Kaplan-Meier survival curves and fit discrete-time hazard regression to quantify how experimental factors shift failure risk over time. Then, to analyze rationales and language patterns associated with failure, we build a seven-category taxonomy from 13,780 deliberation traces from agents who choose to terminate the episode, using an LLM-assisted labeling paired with human audit (kappa=0.83). Rationale profiles change systematically with time and context: early failures are more impulse-driven, later failures more fatigue- and cost-benefit-framed, while public settings increase norm-oriented justifications. We also find a deliberation-inconsistency association: among failures, longer deliberation correlates with higher rates of intra-rationale contradiction (simultaneous pro-delay and pro-claim statements), challenging the assumption that more reasoning text implies greater consistency. Together, the survival and rationale analyses reveal distinct temporal reliability regimes and model-specific "failure fingerprints", offering an evaluation lens for diagnosing inconsistency in multi-turn agent behavior.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 15 of its 32 references that are in the map. First posted on arXiv 26 Aug 2026.
Builds on
15 of 32 references.
Closest ancestors · 2
Same direction, newest first
- 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents24 Jan
- 2023AgentBench: Evaluating LLMs as Agents7 Aug
Other references · 13
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
- 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
- 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
- 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
- 2025LLMs Get Lost In Multi-Turn Conversationcited by 429
- 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
- 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 297
- 2022Thinking Fast and Slow in Large Language Modelscited by 137
- 2025Evaluating Step-by-step Reasoning Traces: A Surveycited by 63
- 2024LLM economicus? Mapping the Behavioral Biases of LLMs via Utility Theorycited by 47
- 2025Measuring Chain of Thought Faithfulness by Unlearning Reasoning Stepscited by 47
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Delay-of-Gratification as a Multi-Agent Survival Micro-benchmark for Long-Horizon LLMs: Social Exposure, Personas, and Tool Use Budgets
- 2026Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents
- 2026ReLiveGym: Evaluating Long-Lived Agents over Weeks of Replayed Reality
- 2026Survive at All Costs: Exploring LLM's Risky Behaviors under Survival Pressure
- 2026Silent Failure in LLM Agent Systems: The Entropy Principle and the Inevitable Disorder of Autonomous Agents
- 2025Check Yourself Before You Wreck Yourself: Selectively Quitting Improves LLM Agent Safety
- 2026Agent Drift: Quantifying Behavioral Degradation in Multi-Agent LLM Systems Over Extended Interactions
- 2025How Do LLMs Fail In Agentic Scenarios? A Qualitative Analysis of Success and Failure Scenarios of Various LLMs in Agentic Simulations