# Beyond IVR: Benchmarking Customer Support LLM Agents for Business-Adherence

arXiv: https://arxiv.org/abs/2601.00596 · HTML: https://mlsanity.com/paper/2601.00596/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 6 of its 60 references that are in the map.
- First posted on arXiv 2 Jan 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 6 of 6)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents](https://mlsanity.com/paper/2407.18901/) (2024)
- [GPT-4o: Visual perception performance of multimodal large language models in piglet activity understanding](https://mlsanity.com/paper/2406.09781/) (2024)

## Built on by (most cited first, 11 of 11)

- [CompliBench: Benchmarking LLM Judges for Compliance Violation Detection in Dialogue Systems](https://mlsanity.com/paper/2604.12312/) (2026)
- [From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents](https://mlsanity.com/paper/2605.15104/) (2026)
- [PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM Agents](https://mlsanity.com/paper/2608.19861/) (2026)
- [Declarative Skills for AI Agents in Knowledge-Grounded Tool-Use Workflows](https://mlsanity.com/paper/2606.06923/) (2026)
- [HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following](https://mlsanity.com/paper/2607.25398/) (2026)
- [ττ-Bench: An Environment for End-To-End, Realistic Agent Construction](https://mlsanity.com/paper/2609.04611/) (2026)
- [COVENANT: Natural-Language Workflow Compilation for Aligned Agent Execution](https://mlsanity.com/paper/2607.25400/) (2026)
- [No Action Without a NOD: A Heterogeneous Multi-Agent Architecture for Reliable Service Agents](https://mlsanity.com/paper/2605.12240/) (2026)
- [Long SKILL Compliance as Logical Reasoning: Closure-Grounded Detection with Scaling-Guided On-Policy Distillation](https://mlsanity.com/paper/2608.08146/) (2026)
- [STAGE: Stateful Translation to Agentic Graph Execution with Policy-Scoped Context and Deterministic Control](https://mlsanity.com/paper/2608.22538/) (2026)
- [Policy Loopholes in Agent Evaluation: When Policy Ambiguity Masquerades as Agent Error](https://mlsanity.com/paper/2609.14400/) (2026)

## Abstract (arXiv)

Traditional customer support systems, such as Interactive Voice Response (IVR), rely on rigid scripts and lack the flexibility required for handling complex, policy-driven tasks. While large language model (LLM) agents offer a promising alternative, evaluating their ability to act in accordance with business rules and real-world support workflows remains an open challenge. Existing benchmarks primarily focus on tool usage or task completion, overlooking an agent's capacity to adhere to multi-step policies, navigate task dependencies, and remain robust to unpredictable user or environment behavior. In this work, we introduce JourneyBench, a benchmark designed to assess policy-aware agents in customer support. JourneyBench leverages graph representations to generate diverse, realistic support scenarios and proposes the User Journey Coverage Score, a novel metric to measure policy adherence. We evaluate multiple state-of-the-art LLMs using two agent designs: a Static-Prompt Agent (SPA) and a Dynamic-Prompt Agent (DPA) that explicitly models policy control. Across 703 conversations in three domains, we show that DPA significantly boosts policy adherence, even allowing smaller models like GPT-4o-mini to outperform more capable ones like GPT-4o. Our findings demonstrate the importance of structured orchestration and establish JourneyBench as a critical resource to advance AI-driven customer support beyond IVR-era limitations.