# TurnWise: The Gap between Single- and Multi-turn Language Model Capabilities

arXiv: https://arxiv.org/abs/2603.16759 · HTML: https://mlsanity.com/paper/2603.16759/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 19 references that are in the map.
- First posted on arXiv 17 Mar 2026.

## Builds on (most cited first, 15 of 15)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues](https://mlsanity.com/paper/2402.14762/) (2024)
- [MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs](https://mlsanity.com/paper/2501.17399/) (2025)
- [APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay](https://mlsanity.com/paper/2504.03601/) (2025)
- [MultiWOZ 2.4: A Multi-Domain Task-Oriented Dialogue Dataset with Essential Annotation Corrections to Improve State Tracking Evaluation](https://mlsanity.com/paper/2104.00773/) (2021)
- [Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk](https://mlsanity.com/paper/2401.05033/) (2024)
- [ToolDial: Multi-turn Dialogue Generation Method for Tool-Augmented Language Models](https://mlsanity.com/paper/2503.00564/) (2025)
- [The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gains](https://mlsanity.com/paper/2507.06187/) (2025)
- [Enhancing Dialogue State Tracking Models through LLM-backed User-Agents Simulation](https://mlsanity.com/paper/2405.13037/) (2024)
- [Goal Alignment in LLM-Based User Simulators for Conversational AI](https://mlsanity.com/paper/2507.20152/) (2025)
- [Self-Directed Synthetic Dialogues and Revisions Technical Report](https://mlsanity.com/paper/2407.18421/) (2024)
- [Another Turn, Better Output? A Turn-Wise Analysis of Iterative LLM Prompting](https://mlsanity.com/paper/2509.06770/) (2025)

## Built on by (most cited first, 3 of 3)

- [Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction](https://mlsanity.com/paper/2604.04325/) (2026)
- [Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challenges](https://mlsanity.com/paper/2608.17605/) (2026)
- [Inverse Turing Bench: Evaluating Language Models as Judges of Human vs. AI Dialogue](https://mlsanity.com/paper/2606.21844/) (2026)

## Abstract (arXiv)

Multi-turn conversations are a common and critical mode of language model interaction. However, current open training and evaluation data focus on single-turn settings, failing to capture the additional dimension of these longer interactions. To understand this multi-/single-turn gap, we first introduce a new benchmark, TurnWiseEval, for multi-turn capabilities that is directly comparable to single-turn chat evaluation. Our evaluation isolates multi-turn specific conversational ability through pairwise comparison to equivalent single-turn settings. We additionally introduce our synthetic multi-turn data pipeline TurnWiseData which allows the scalable generation of multi-turn training data. Our experiments with Olmo 3 show that training with multi-turn data is vital to achieving strong multi-turn chat performance, and that including as little as 10k multi-turn conversations during post-training can lead to a 12% improvement on TurnWiseEval.