# TOD-ProcBench: Benchmarking Complex Instruction-Following in Task-Oriented Dialogues

arXiv: https://arxiv.org/abs/2511.15976 · HTML: https://mlsanity.com/paper/2511.15976/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 25 references that are in the map.
- First posted on arXiv 20 Nov 2025.

## Builds on (most cited first, 16 of 16)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset](https://mlsanity.com/paper/1909.05855/) (2019)
- [Benchmarking Complex Instruction-Following with Multiple Constraints Composition](https://mlsanity.com/paper/2407.03978/) (2024)
- [Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.00783/) (2021)
- [Semantically Conditioned Dialog Response Generation via Hierarchical Disentangled Self-Attention](https://mlsanity.com/paper/1905.12866/) (2019)
- [STAR: A Schema-Guided Dialog Dataset for Transfer Learning](https://mlsanity.com/paper/2010.11853/) (2020)
- [Conditional Generation with a Question-Answering Blueprint](https://mlsanity.com/paper/2207.00397/) (2022)
- [Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2402.13374/) (2024)
- [SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints](https://mlsanity.com/paper/2503.08669/) (2025)
- [SOP-Bench: Complex Industrial SOPs for Evaluating LLM Agents](https://mlsanity.com/paper/2506.08119/) (2025)
- [Chain of Thought Prompting Elicits Knowledge Augmentation](https://mlsanity.com/paper/2307.01640/) (2023)
- [FLAP: Flow-Adhering Planning with Constrained Decoding in LLMs](https://mlsanity.com/paper/2403.05766/) (2024)
- [Personalizing Task-oriented Dialog Systems via Zero-shot Generalizable Reward Function](https://mlsanity.com/paper/2303.13797/) (2023)
- [Improving Generalization in Task-oriented Dialogues with Workflows and Action Plans](https://mlsanity.com/paper/2306.01729/) (2023)
- [Chain of Thought Explanation for Dialogue State Tracking](https://mlsanity.com/paper/2403.04656/) (2024)
- [Workflow-Guided Response Generation for Task-Oriented Dialogue](https://mlsanity.com/paper/2311.08300/) (2023)

## Built on by (most cited first, 2 of 2)

- [Near-Miss: Latent Policy Failure Detection in Agentic Workflows](https://mlsanity.com/paper/2603.29665/) (2026)
- [Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challenges](https://mlsanity.com/paper/2608.17605/) (2026)

## Abstract (arXiv)

In real-world task-oriented dialogue (TOD) settings, agents are required to strictly adhere to complex instructions while conducting multi-turn conversations with customers. These instructions are typically presented in natural language format and include general guidelines and step-by-step procedures with complex constraints. Existing TOD benchmarks often oversimplify the complex nature of these instructions by reducing them to simple schemas composed of intents, slots, and API call configurations. To address this gap and systematically benchmark LLMs' instruction-following capabilities, we propose TOD-ProcBench, a challenging benchmark featuring complex process instructions with intricate, fine-grained constraints that evaluates various LLMs' abilities to understand and follow instructions in multi-turn TODs. Our benchmark dataset comprises instruction documents derived from the high-quality ABCD dataset with corresponding conversations under human quality control. We formulate fine-grained constraints and action procedures as multi-level condition-action instruction statements. We design three tasks to comprehensively benchmark LLMs' complex instruction-following capabilities in multi-turn TODs. Task 1 evaluates how LLMs retrieve the most relevant statement from a complex instruction and predict the corresponding next action. In Task 2, we synthesize instruction-violating responses by injecting inconsistencies and manipulating the original instructions, and then we analyze how effectively LLMs can identify instruction-violating responses. Task 3 investigates LLMs' abilities in conditional generation of instruction-following responses based on the original complex instructions. Additionally, we conduct studies on the impact of multilingual settings and different instruction text formats on compliance performance. We release our benchmark under the Llama 3.3 Community License Agreement.