# One Battle After Another: Probing LLMs' Limits on Multi-Turn Instruction Following with a Benchmark Evolving Framework

arXiv: https://arxiv.org/abs/2511.03508 · HTML: https://mlsanity.com/paper/2511.03508/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 42 references that are in the map.
- First posted on arXiv 5 Nov 2025.

## Builds on (most cited first, 22 of 22)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [Self-Preference Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2410.21819/) (2024)
- [Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions](https://mlsanity.com/paper/2507.05257/) (2025)
- [MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues](https://mlsanity.com/paper/2402.14762/) (2024)
- [MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs](https://mlsanity.com/paper/2501.17399/) (2025)
- [Benchmarking Complex Instruction-Following with Multiple Constraints Composition](https://mlsanity.com/paper/2407.03978/) (2024)
- [Can Large Language Models Understand Real-World Complex Instructions?](https://mlsanity.com/paper/2309.09150/) (2023)
- [CFBench: A Comprehensive Constraints-Following Benchmark for LLMs](https://mlsanity.com/paper/2408.01122/) (2024)
- [Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussions](https://mlsanity.com/paper/2405.20267/) (2024)
- [FairMT-Bench: Benchmarking Fairness for Multi-turn Dialogue in Conversational LLMs](https://mlsanity.com/paper/2410.19317/) (2024)
- [Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2508.06709/) (2025)
- [StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction Following](https://mlsanity.com/paper/2502.14494/) (2025)
- [How Reliable is Your Simulator? Analysis on the Limitations of Current LLM-based User Simulators for Conversational Recommendation](https://mlsanity.com/paper/2403.16416/) (2024)
- [The SIFo Benchmark: Investigating the Sequential Instruction Following Ability of Large Language Models](https://mlsanity.com/paper/2406.19999/) (2024)
- [CodeIF-Bench: Evaluating Instruction-Following Capabilities of Large Language Models in Interactive Code Generation](https://mlsanity.com/paper/2503.22688/) (2025)
- [MTR-Bench: A Comprehensive Benchmark for Multi-Turn Reasoning Evaluation](https://mlsanity.com/paper/2505.17123/) (2025)
- [Can Language Models Follow Multiple Turns of Entangled Instructions?](https://mlsanity.com/paper/2503.13222/) (2025)
- [SimulBench: Evaluating Language Models with Creative Simulation Tasks](https://mlsanity.com/paper/2409.07641/) (2024)
- [From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions](https://mlsanity.com/paper/2502.13791/) (2025)
- [Meta-optimized Joint Generative and Contrastive Learning for Sequential Recommendation](https://mlsanity.com/paper/2310.13925/) (2023)
- [DELRec: Distilling Sequential Pattern to Enhance LLMs-based Sequential Recommendation](https://mlsanity.com/paper/2406.11156/) (2024)

## Built on by (most cited first, 9 of 9)

- [When Attention Closes: How LLMs Lose the Thread in Multi-Turn Interaction](https://mlsanity.com/paper/2605.12922/) (2026)
- [Where do LLMs Fall Short in CBT-Guided Affective Reasoning?](https://mlsanity.com/paper/2607.02885/) (2026)
- [Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging](https://mlsanity.com/paper/2607.10428/) (2026)
- [Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challenges](https://mlsanity.com/paper/2608.17605/) (2026)
- [The Unreliable Progress Bar: Can LLM Agents Reliably Report Task Progress Throughout Execution?](https://mlsanity.com/paper/2609.08589/) (2026)
- [Hy-MultiTurn: A Six-Dimensional Benchmark for Deep Multi-Turn Dialogue Understanding](https://mlsanity.com/paper/2607.29196/) (2026)
- [Dead text or binding clause? Measuring and restoring constraint influence in black-box LLM dialogues](https://mlsanity.com/paper/2608.12599/) (2026)
- [When Users Change Their Minds: Measuring and Repairing Intent Drift in LLM Agents](https://mlsanity.com/paper/2609.32520/) (2026)
- [The Effects of Incremental Instruction Delivery on Language-Model Creative Writing](https://mlsanity.com/paper/2609.33738/) (2026)

## Abstract (arXiv)

Evaluating LLMs' instruction-following ability in multi-topic dialogues is essential yet challenging. Existing benchmarks are limited to a fixed number of turns, susceptible to saturation and failing to account for users' interactive experience. In this work, we propose a novel framework featuring a three-layer tracking mechanism and a query synthesis agent to mimic sequential user behaviors. Grounded in Flow Theory, we introduce process-centric metrics and terminate a conversational evaluation only upon exhausting user patience. Leveraging this framework, we present EvolIF, an evolving benchmark covering 12 constraint groups. Our analysis reveals deficiencies in failure recovery and fine-grained instruction following, with performance stratification becoming evident as conversational depth increases. GPT-5 demonstrates the most sustained resilience, maintaining a 66.40% robustness score, outperforming Gemini-3-Pro by 5.59%, while other models lag behind. Data and code will be released at https://github.com/JiaQiSJTU/EvolIF.