# Evaluating, Synthesizing, and Enhancing for Customer Support Conversation

arXiv: https://arxiv.org/abs/2508.04423 · HTML: https://mlsanity.com/paper/2508.04423/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 30 references that are in the map.
- First posted on arXiv 6 Aug 2025.

## Builds on (most cited first, 16 of 16)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Towards Emotional Support Dialog Systems](https://mlsanity.com/paper/2106.01144/) (2021)
- [A Network-based End-to-End Trainable Task-oriented Dialogue System](https://mlsanity.com/paper/1604.04562/) (2016)
- [Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset](https://mlsanity.com/paper/1909.05358/) (2019)
- [Frames: A Corpus for Adding Memory to Goal-Oriented Dialogue Systems](https://mlsanity.com/paper/1704.00057/) (2017)
- [From Role-Play to Drama-Interaction: An LLM Solution](https://mlsanity.com/paper/2405.14231/) (2024)
- [OpenCharacter: Training Customizable Role-Playing LLMs with Large-Scale Synthetic Personas](https://mlsanity.com/paper/2501.15427/) (2025)
- [SweetieChat: A Strategy-Enhanced Role-playing Framework for Diverse Scenarios Handling Emotional Support Agent](https://mlsanity.com/paper/2412.08389/) (2024)
- ["How May I Help You?": Modeling Twitter Customer Service Conversations Using Fine-Grained Dialogue Acts](https://mlsanity.com/paper/1709.05413/) (2017)
- [NatCS: Eliciting Natural Customer Support Dialogues](https://mlsanity.com/paper/2305.03007/) (2023)

## Built on by (most cited first, 7 of 7)

- [FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol](https://mlsanity.com/paper/2603.24943/) (2026)
- [SEAD: Self-Evolving Agent for Multi-Turn Service Dialogue](https://mlsanity.com/paper/2602.03548/) (2026)
- [CirrusBench: Evaluating LLM-based Agents Beyond Correctness in Real-World Cloud Service Environments](https://mlsanity.com/paper/2603.28569/) (2026)
- [Reinforcing Real-world Service Agents: Balancing Utility and Cost in Task-oriented Dialogue](https://mlsanity.com/paper/2602.22697/) (2026)
- [IB-RL: Isolated Bilateral Reinforcement Learning for Strategic Dialogue Agents](https://mlsanity.com/paper/2608.06735/) (2026)
- [TRACER: Trajectory-Aligned Learning for Multi-Turn User Simulation](https://mlsanity.com/paper/2609.28690/) (2026)
- [Chinese-Jev: Bringing System One Model to Chinese-Language Tasks](https://mlsanity.com/paper/2609.36965/) (2026)

## Abstract (arXiv)

Effective customer support requires not only accurate problem solving but also structured and empathetic communication aligned with professional standards. However, existing dialogue datasets often lack strategic guidance, and real-world service data is difficult to access and annotate. To address this, we introduce the task of Customer Support Conversation (CSC), aimed at training customer service agents to respond using well-defined support strategies. We propose a structured CSC framework grounded in COPC guidelines, defining five conversational stages and twelve strategies to guide high-quality interactions. Based on this, we construct CSConv, an evaluation dataset of 1,855 real-world customer-agent conversations rewritten using LLMs to reflect deliberate strategy use, and annotated accordingly. Additionally, we develop a role-playing approach that simulates strategy-rich conversations using LLM-powered roles aligned with the CSC framework, resulting in the training dataset RoleCS. Experiments show that fine-tuning strong LLMs on RoleCS significantly improves their ability to generate high-quality, strategy-aligned responses on CSConv. Human evaluations further confirm gains in problem resolution. All code and data will be made publicly available at https://github.com/aliyun/qwen-dianjin.