mlsanityData to 7 Oct 2026

ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions

arXiv 2601.06112 ↗First posted 3 Jan 2026cs.AIcited by 31

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Existing benchmarks for tool-using LLM agents primarily report single-run success rates and miss reliability properties required in production. We introduce ReliabilityBench, a benchmark for evaluating agent reliability across three dimensions: (i) consistency under repeated execution using passk, (ii) robustness to semantically equivalent task perturbations at intensity ε, and (iii) fault tolerance under controlled tool/API failures at intensity λ. ReliabilityBench contributes a unified reliability surface R(k,ε,λ), textitaction metamorphic relations that define correctness via end-state equivalence rather than text similarity, and a chaos-engineering-style fault injection framework (timeouts, rate limits, partial responses, schema drift). We evaluate two models (Gemini 2.0 Flash, GPT-4o) and two agent architectures (ReAct, Reflexion) across four domains (scheduling, travel, customer support, e-commerce) over 1,280 episodes. Perturbations alone reduce success from 96.9% at ε=0 to 88.1% at ε=0.2. Rate limiting is the most damaging fault in ablations. ReAct is more robust than Reflexion under combined stress, and Gemini 2.0 Flash achieves comparable reliability to GPT-4o at much lower cost. ReliabilityBench provides a systematic framework for assessing production readiness of LLM agents.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 16 of its 36 references that are in the map. First posted on arXiv 3 Jan 2026.

Builds on

16 of 36 references.

References · 16

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  6. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  7. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  8. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  9. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  10. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  11. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  12. 2017DeepXplore: Automated Whitebox Testing of Deep Learning Systemscited by 307
4 more
  1. 2024StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Modelscited by 129
  2. 2019Taxonomy of Real Faults in Deep Learning Systemscited by 90
  3. 2020Metamorphic Testing: A New Approach for Generating Next Test Casescited by 71
  4. 2017Chaos Engineeringcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 31.

31 papers.

Sort:

By area:

  1. 2026When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agentscited by 15
  2. 2026From Question Answering to Task Completion: A Survey on Agent System and Harness Designcited by 9
  3. 2026Self-Healing Agentic Orchestrators for Reliable Tool-Augmented Large Language Model Systemscited by 7
  4. 2026Compiling Agentic Workflows into LLM Weights: Near-Frontier Quality at Two Orders of Magnitude Less Costcited by 4
  5. 2026When Agents Disagree With Themselves: Behavioral Consistency as an Uncertainty Signal for LLM Agentscited by 2
  6. 2026In-Context Prompting Obsoletes Agent Orchestration for Procedural Taskscited by 2
  7. 2026A Self-Healing Framework for Reliable LLM-Based Autonomous Agentscited by 2
  8. 2026How Much Coordination Gain Is Real? A Paired Noise-Floor Protocol for Multi-Agent LLM Benchmarkscited by 2
  9. 2026The Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent Behaviorcited by 2
  10. 2026Latency-Quality Routing for Functionally Equivalent Tools in LLM Agentscited by 1
  11. 2026PlanningBench: Generating Scalable and Verifiable Planning Data for Evaluating and Training Large Language Modelscited by 1
  12. 2026Toward User-Conditioned Evaluation of Personal LLM Agents under Temporal Interventionscited by 1
19 more
  1. 2026Tangent: An Empirical Study of Testing Practices for LLM-Based Agent Applicationscited by 1
  2. 2026SkillSentry: Reliable Skill Execution for LLM Agents via Runtime Assurancecited by 1
  3. 2026When Agentic Executions Fail: Detecting and Localizing Runtime Faults from Telemetrycited by 1
  4. 2026Cost-Utility Alignment in LLM Agent Trajectories:Profiling,Attribution,Diagnosis,Adaptation,and Evaluationcited by 1
  5. 2026Toward Self-Evolution-Ready Workflow Harnesses: A Reversible Migration Path and Convertibility Taxonomy for Expert LLM Pipelinescited by 0
  6. 2026When the Database Fails: Prompting LLM Dialogue Agents for Safe Recovery in Task-Oriented Dialoguecited by 0
  7. 2026Beyond Static Evaluation: Building Simulation Environments for Scalable Agentic Reinforcement Learningcited by 0
  8. 2026Towards an Agent Operating System - Lessons from Classical and Cloud OScited by 0
  9. 2026LayerRAG-Bench: A Cross-Layer Reliability Benchmark for Agentic Retrieval-Augmented Generationcited by 0
  10. 2026Retry, Switch, or Abstain? Learning Strategy-Aware Tool-Use Policies via Controlled Error Injectioncited by 0
  11. 2026No Task Fails Every Time: Why One-Shot Audits Are Structurally Blind to Agent Damagecited by 0
  12. 2026Towards Risk-free AI Agent Deploymentcited by 0
  13. 2026SilentProbe: Measuring Silent Failure in Production APIs Used as Agent Toolscited by 0
  14. 2026Same Patient, Different Order: Action-Level Reliability of Clinical LLM Agents Under Repeated Runscited by 0
  15. 2026When Hiring Becomes Agent-Mediated: Evaluating Access and Recurrence in Two-Agent R'esum'e Screeningcited by 0
  16. 2026Where Does Exactly-Once Live? Model, Harness, and Tool-Contract Effects on Duplicate Side Effects in LLM Agentscited by 0
  17. 2026Failure-Transparent Agents: Benchmarking Post-Failure Reporting in Tool-Using Language Modelscited by 0
  18. 2026Constructing Challenging Browser-Use Tasks by Controlled Environment Interventionscited by 0
  19. 2026Loud Failures, Quiet Failures: Fault Detection and Recovery in Tool-Using Language Model Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026When Agents Fail to Act: A Diagnostic Framework for Tool Invocation Reliability in Multi-Agent LLM Systems
  2. 2026Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents
  3. 2026Verified Tool Calls Improve LLM Agent Reliability Under Non-Atomic Failures
  4. 2026Beyond Function Calling: Benchmarking Tool-Using Agents under Tool-Environment Unreliability
  5. 2026AgentFixer: From Failure Detection to Fix Recommendations in LLM Agentic Systems
  6. 2026How Consistent Are LLM Agents? Measuring Behavioral Reproducibility in Multi-Step Tool-Calling Pipelines
  7. 2026The Unreliable Progress Bar: Can LLM Agents Reliably Report Task Progress Throughout Execution?
  8. 2026The Six Sigma Agent: Achieving Enterprise-Grade Reliability in LLM Systems Through Consensus-Driven Decomposed Execution