# Turning Conversations into Workflows: A Framework to Extract and Evaluate Dialog Workflows for Service AI Agents

arXiv: https://arxiv.org/abs/2502.17321 · HTML: https://mlsanity.com/paper/2502.17321/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 42 references that are in the map.
- First posted on arXiv 24 Feb 2025.

## Builds on (most cited first, 28 of 28)

- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization](https://mlsanity.com/paper/2112.08542/) (2021)
- [Chain of Agents: Large Language Models Collaborating on Long-Context Tasks](https://mlsanity.com/paper/2406.02818/) (2024)
- [WikiHow: A Large Scale Text Summarization Dataset](https://mlsanity.com/paper/1810.09305/) (2018)
- [Universal Self-Consistency for Large Language Model Generation](https://mlsanity.com/paper/2311.17311/) (2023)
- [Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.00783/) (2021)
- [STAR: A Schema-Guided Dialog Dataset for Transfer Learning](https://mlsanity.com/paper/2010.11853/) (2020)
- [Debate-to-Write: A Persona-Driven Multi-Agent Framework for Diverse Argument Generation](https://mlsanity.com/paper/2406.19643/) (2024)
- [Modelling Hierarchical Structure between Dialogue Policy and Natural Language Generator with Option Framework for Task-oriented Dialogue System](https://mlsanity.com/paper/2006.06814/) (2020)
- [DialoKG: Knowledge-Structure Aware Task-Oriented Dialogue Generation](https://mlsanity.com/paper/2204.09149/) (2022)
- [Unsupervised Dialog Structure Learning](https://mlsanity.com/paper/1904.03736/) (2019)
- [Schema-Guided Paradigm for Zero-Shot Dialog](https://mlsanity.com/paper/2106.07056/) (2021)
- [Workflow Discovery from Dialogues in the Low Data Regime](https://mlsanity.com/paper/2205.11690/) (2022)
- [A Benchmark for Structured Procedural Knowledge Extraction from Cooking Videos](https://mlsanity.com/paper/2005.00706/) (2020)
- [AnyTOD: A Programmable Task-Oriented Dialog System](https://mlsanity.com/paper/2212.09939/) (2022)
- [LLM Based Multi-Agent Generation of Semi-structured Documents from Semantic Templates in the Public Administration Domain](https://mlsanity.com/paper/2402.14871/) (2024)
- [Structure Extraction in Task-Oriented Dialogues with Slot Clustering](https://mlsanity.com/paper/2203.00073/) (2022)
- [CTRLStruct: Dialogue Structure Learning for Open-Domain Response Generation](https://mlsanity.com/paper/2303.01094/) (2023)
- [Dialog2Flow: Pre-training Soft-Contrastive Action-Driven Sentence Embeddings for Automatic Dialog Flow Extraction](https://mlsanity.com/paper/2410.18481/) (2024)
- [Conversation Routines: A Prompt Engineering Framework for Task-Oriented Dialog Systems](https://mlsanity.com/paper/2501.11613/) (2025)
- [Unsupervised Learning of Hierarchical Conversation Structure](https://mlsanity.com/paper/2205.12244/) (2022)
- [Using Domain Knowledge to Guide Dialog Structure Induction via Neural Probabilistic Soft Logic](https://mlsanity.com/paper/2403.17853/) (2024)
- [Discovering Dialog Structure Graph for Open-Domain Dialog Generation](https://mlsanity.com/paper/2012.15543/) (2020)
- [DSBERT:Unsupervised Dialogue Structure learning with BERT](https://mlsanity.com/paper/2111.04933/) (2021)
- [Improving Generalization in Task-oriented Dialogues with Workflows and Action Plans](https://mlsanity.com/paper/2306.01729/) (2023)
- [TSCAN : Dialog Structure discovery using SCAN](https://mlsanity.com/paper/2107.06426/) (2021)

## Built on by (most cited first, 5 of 5)

- [Generative Induction of Dialogue Task Schemas with Streaming Refinement and Simulated Interactions](https://mlsanity.com/paper/2504.18474/) (2025)
- [METRO: Towards Strategy Induction from Expert Dialogue Transcripts for Non-collaborative Dialogues](https://mlsanity.com/paper/2604.11427/) (2026)
- [Text-to-SQL Task-oriented Dialogue Ontology Construction](https://mlsanity.com/paper/2507.23358/) (2025)
- [WISE-Flow: Workflow-Induced Structured Experience for Self-Evolving Conversational Service Agents](https://mlsanity.com/paper/2601.08158/) (2026)
- [Staged Linguistic Seeding: Grounded Query Expansion for Verified-Unit QA in AI Contact Centers](https://mlsanity.com/paper/2609.00844/) (2026)

## Abstract (arXiv)

Automated service agents require well-structured workflows to provide consistent and accurate responses to customer queries. However, these workflows are often undocumented, and their automatic extraction from conversations remains unexplored. In this work, we present a novel framework for extracting and evaluating dialog workflows from historical interactions. Our extraction process consists of two key stages: (1) a retrieval step to select relevant conversations based on key procedural elements, and (2) a structured workflow generation process using a question-answer-based chain-of-thought (QA-CoT) prompting. To comprehensively assess the quality of extracted workflows, we introduce an automated agent and customer bots simulation framework that measures their effectiveness in resolving customer issues. Extensive experiments on the ABCD and SynthABCD datasets demonstrate that our QA-CoT technique improves workflow extraction by 12.16% in average macro accuracy over the baseline. Moreover, our evaluation method closely aligns with human assessments, providing a reliable and scalable framework for future research.