# StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction Following

arXiv: https://arxiv.org/abs/2502.14494 · HTML: https://mlsanity.com/paper/2502.14494/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 22 references that are in the map.
- First posted on arXiv 20 Feb 2025.

## Builds on (most cited first, 14 of 14)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset](https://mlsanity.com/paper/2309.11998/) (2023)
- [MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback](https://mlsanity.com/paper/2309.10691/) (2023)
- [MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues](https://mlsanity.com/paper/2402.14762/) (2024)
- [Benchmarking Complex Instruction-Following with Multiple Constraints Composition](https://mlsanity.com/paper/2407.03978/) (2024)
- [Can Large Language Models Understand Real-World Complex Instructions?](https://mlsanity.com/paper/2309.09150/) (2023)
- [CFBench: A Comprehensive Constraints-Following Benchmark for LLMs](https://mlsanity.com/paper/2408.01122/) (2024)
- [BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialogues](https://mlsanity.com/paper/2310.13650/) (2023)

## Built on by (most cited first, 29 of 29)

- [Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Models](https://mlsanity.com/paper/2505.14810/) (2025)
- [AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition](https://mlsanity.com/paper/2602.11348/) (2026)
- [RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Following](https://mlsanity.com/paper/2603.25133/) (2026)
- [Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models](https://mlsanity.com/paper/2506.01413/) (2025)
- [IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation](https://mlsanity.com/paper/2603.04738/) (2026)
- [One Battle After Another: Probing LLMs' Limits on Multi-Turn Instruction Following with a Benchmark Evolving Framework](https://mlsanity.com/paper/2511.03508/) (2025)
- [Revisiting the Reliability of Language Models in Instruction-Following](https://mlsanity.com/paper/2512.14754/) (2025)
- [Learning to Act under Noise: Enhancing Agent Robustness via Noisy Environments](https://mlsanity.com/paper/2605.27209/) (2026)
- [SOP-Maze: Evaluating Large Language Models on Complicated Business Standard Operating Procedures](https://mlsanity.com/paper/2510.08942/) (2025)
- [Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction](https://mlsanity.com/paper/2604.04325/) (2026)
- [Meeseeks: A Feedback-Driven, Iterative Self-Correction Benchmark evaluating LLMs' Instruction Following Capability](https://mlsanity.com/paper/2504.21625/) (2025)
- [Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logs](https://mlsanity.com/paper/2510.25441/) (2025)
- [CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training](https://mlsanity.com/paper/2603.06610/) (2026)
- [Evaluating Ill-Defined Tasks in Large Language Models](https://mlsanity.com/paper/2603.17067/) (2026)
- [Useless but Safe? Benchmarking Utility Recovery with User Intent Clarification in Multi-Turn Conversations](https://mlsanity.com/paper/2604.27093/) (2026)
- [Same Evidence, Different Answers: Canonical-Context On-Policy Distillation for Multi-Turn Language Models](https://mlsanity.com/paper/2605.30251/) (2026)
- [KnowMT-Bench: Benchmarking Knowledge-Intensive Long-Form Question Answering in Multi-Turn Dialogues](https://mlsanity.com/paper/2509.21856/) (2025)
- [Deconstructing Instruction-Following: A New Benchmark for Granular Evaluation of Large Language Model Instruction Compliance Abilities](https://mlsanity.com/paper/2601.18554/) (2026)
- [Quantifying Conversational Reliability of Large Language Models under Multi-Turn Interaction](https://mlsanity.com/paper/2603.01423/) (2026)
- [IFHierBench: Hierarchical Instruction Following for Large Language Models](https://mlsanity.com/paper/2607.27912/) (2026)
- [Guided Decoding and Its Critical Role in Retrieval-Augmented Generation](https://mlsanity.com/paper/2509.06631/) (2025)
- [The Atomic Instruction Gap: Instruction-Tuned LLMs Struggle with Simple, Self-Contained Directives](https://mlsanity.com/paper/2510.17388/) (2025)
- [Style Amnesia: Investigating Speaking Style Degradation and Mitigation in Multi-Turn Spoken Language Models](https://mlsanity.com/paper/2512.23578/) (2025)
- [Hy-MultiTurn: A Six-Dimensional Benchmark for Deep Multi-Turn Dialogue Understanding](https://mlsanity.com/paper/2607.29196/) (2026)
- [Video-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding Scenarios](https://mlsanity.com/paper/2608.25529/) (2026)
- [Towards Proactive Detection of User-Side Implicit Conflicts in Human-LLM Dialogue](https://mlsanity.com/paper/2609.19155/) (2026)
- [Clarification Is Not Correction: LLMs Fail to Let Go](https://mlsanity.com/paper/2609.25337/) (2026)
- [The Effects of Incremental Instruction Delivery on Language-Model Creative Writing](https://mlsanity.com/paper/2609.33738/) (2026)
- [A GHOST in Long-Horizon Agents: Governance Hazard from Overlooked Safety Constraints across Turns](https://mlsanity.com/paper/2610.02664/) (2026)

## Abstract (arXiv)

Multi-turn instruction following capability constitutes a core competency of large language models (LLMs) in real-world applications. Existing evaluation benchmarks predominantly focus on fine-grained constraint satisfaction and domain-specific capability assessment, yet overlook the crucial structural dependencies between dialogue turns that distinguish multi-turn from single-turn interactions. These structural dependencies not only reflect user intent but also establish an essential second dimension for the instruction following evaluation beyond constraint satisfaction. To address this gap, we propose StructFlowBench, a multi-turn instruction following benchmark with structural flow modeling. The benchmark defines an innovative structural flow framework with six fundamental inter-turn relationships. These relationships introduce novel structural constraints for model evaluation and also serve as generation parameters for creating customized dialogue flows tailored to specific scenarios. Adopting established LLM-based automatic evaluation methodologies, we conduct systematic evaluations of 13 leading open-source and closed-source LLMs. Experimental results reveal significant deficiencies in current models' comprehension of multi-turn dialogue structures. The code is available at https://github.com/MLGroupJLU/StructFlowBench.