# TaskDiff: A Similarity Metric for Task-Oriented Conversations

arXiv: https://arxiv.org/abs/2310.15298 · HTML: https://mlsanity.com/paper/2310.15298/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 31 references that are in the map.
- First posted on arXiv 23 Oct 2023.

## Builds on (most cited first, 12 of 12)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset](https://mlsanity.com/paper/1909.05855/) (2019)
- [Task-Oriented Dialogue as Dataflow Synthesis](https://mlsanity.com/paper/2009.11423/) (2020)
- [Hierarchical Optimal Transport for Document Representation](https://mlsanity.com/paper/1906.10827/) (2019)
- [Optimal Transport on Discrete Domains](https://mlsanity.com/paper/1801.07745/) (2018)
- [We've had this conversation before: A Novel Approach to Measuring Dialog Similarity](https://mlsanity.com/paper/2110.05780/) (2021)
- [Combining Textual Content and Structure to Improve Dialog Similarity](https://mlsanity.com/paper/1802.07117/) (2018)

## Built on by (most cited first, 1 of 1)

- [A Similarity Measure for Comparing Conversational Dynamics](https://mlsanity.com/paper/2507.18956/) (2025)

## Abstract (arXiv)

The popularity of conversational digital assistants has resulted in the availability of large amounts of conversational data which can be utilized for improved user experience and personalized response generation. Building these assistants using popular large language models like ChatGPT also require additional emphasis on prompt engineering and evaluation methods. Textual similarity metrics are a key ingredient for such analysis and evaluations. While many similarity metrics have been proposed in the literature, they have not proven effective for task-oriented conversations as they do not take advantage of unique conversational features. To address this gap, we present TaskDiff, a novel conversational similarity metric that utilizes different dialogue components (utterances, intents, and slots) and their distributions to compute similarity. Extensive experimental evaluation of TaskDiff on a benchmark dataset demonstrates its superior performance and improved robustness over other related approaches.