When Better Turns Do Not Make Better Agents: Diagnosing the Gap Between Next-Turn Metrics and Workflow Success
Not on the map. Why?
Abstract (arXiv)
Agent models are frequently evaluated one decision at a time, where the model predicts the next action based on the gold interaction history, which is scored against a reference. We investigate whether improvement under this protocol is predictive of improved autonomous workflow execution. We study pre-SFT and supervised fine-tuned (SFT) Qwen3 models at 4B and 14B parameters and Gemma 3 models at 4B and 12B parameters on multi-turn customer-support workflows. We find that SFT consistently improves text-turn success, and that overall next-turn success increases for every model under gold-history evaluation. However, these improvements do not transfer to autonomous workflow execution. Tool-specific gains also vary across metrics and models. None of the four SFT models succeeds under holistic workflow evaluation, with strict trajectory completion reaching at most 10.4% workflow success. Our results show that next-turn evaluation is not a reliable proxy for workflow success, motivating separate reporting of text quality, local action correctness, tool execution, and end-to-end task completion.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 22 of its 30 references that are in the map. First posted on arXiv 18 Sep 2026.
Builds on
22 of 30 references.
References · 22
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
- 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
- 2024A Survey on LLM-as-a-Judgecited by 1,423
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
- 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 766
- 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
10 more
- 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
- 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 434
- 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 288
- 2024Tiny Titans: Can Smaller Large Language Models Punch Above Their Weight in the Real World for Meeting Summarization?cited by 33
- 2025CONFETTI: Conversational Function-Calling Evaluation Through Turn-Level Interactionscited by 18
- 2025Judging the Judges: Can Large Vision-Language Models Fairly Evaluate Chart Comprehension and Reasoning?cited by 10
- 2023AI Coach Assist: An Automated Approach for Call Recommendation in Contact Centers for Agent Coachingcited by 8
- 2025AI Knowledge Assist: An Automated Approach for the Creation of Knowledge Bases for Conversational AI Agentscited by 5
- 2026From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agentscited by 5
- 2025DACIP-RC: Domain Adaptive Continual Instruction Pre-Training via Reading Comprehension on Business Conversationscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.