# MDDial: A Multi-turn Differential Diagnosis Dialogue Dataset with Reliability Evaluation

arXiv: https://arxiv.org/abs/2308.08147 · HTML: https://mlsanity.com/paper/2308.08147/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 16 Aug 2023.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [A Simple Language Model for Task-Oriented Dialogue](https://mlsanity.com/paper/2005.00796/) (2020)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [Transformers as Soft Reasoners over Language](https://mlsanity.com/paper/2002.05867/) (2020)
- [MedDialog: Two Large-scale Medical Dialogue Datasets](https://mlsanity.com/paper/2004.03329/) (2020)
- [Hello, It's GPT-2 -- How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1907.05774/) (2019)
- [Dialogue State Tracking with a Language Model using Schema-Driven Prompting](https://mlsanity.com/paper/2109.07506/) (2021)
- [Dr.ICL: Demonstration-Retrieved In-context Learning](https://mlsanity.com/paper/2305.14128/) (2023)
- [End-to-End Knowledge-Routed Relational Dialogue System for Automatic Diagnosis](https://mlsanity.com/paper/1901.10623/) (2019)
- [In-Context Learning for Few-Shot Dialogue State Tracking](https://mlsanity.com/paper/2203.08568/) (2022)
- [Improving Biomedical Information Retrieval with Neural Retrievers](https://mlsanity.com/paper/2201.07745/) (2022)
- [In-BoXBART: Get Instructions into Biomedical Multi-Task Learning](https://mlsanity.com/paper/2204.07600/) (2022)
- [Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challenges](https://mlsanity.com/paper/2203.10012/) (2022)
- [BioTABQA: Instruction Learning for Biomedical Table Question Answering](https://mlsanity.com/paper/2207.02419/) (2022)
- [Can Transformers Reason About Effects of Actions?](https://mlsanity.com/paper/2012.09938/) (2020)
- [A Bayesian Approach for Medical Inquiry and Disease Inference in Automated Differential Diagnosis](https://mlsanity.com/paper/2110.08393/) (2021)

## Built on by (most cited first, 10 of 10)

- [If in a Crowdsourced Data Annotation Pipeline, a GPT-4](https://mlsanity.com/paper/2402.16795/) (2024)
- [Assessing Empathy in Large Language Models with Real-World Physician-Patient Interactions](https://mlsanity.com/paper/2405.16402/) (2024)
- [Interpretable Differential Diagnosis with Dual-Inference Large Language Models](https://mlsanity.com/paper/2407.07330/) (2024)
- [Collaborative Rational Speech Act: Pragmatic Reasoning for Multi-Turn Dialog](https://mlsanity.com/paper/2507.14063/) (2025)
- [All Required, In Order: Phase-Level Evaluation for AI-Human Dialogue in Healthcare and Beyond](https://mlsanity.com/paper/2601.08690/) (2026)
- [MedAidDialog: A Multilingual Multi-Turn Medical Dialogue Dataset for Accessible Healthcare](https://mlsanity.com/paper/2603.24132/) (2026)
- [MTDiag: A Multi-Turn Diagnostic Dataset Towards Clinically Meaningful LLM Evaluation](https://mlsanity.com/paper/2608.25085/) (2026)
- [Collaborative Medical Triage under Uncertainty: A Multi-Agent Dynamic Matching Approach](https://mlsanity.com/paper/2507.22504/) (2025)
- [Leveraging Evidence-Guided LLMs to Enhance Trustworthy Depression Diagnosis](https://mlsanity.com/paper/2511.17947/) (2025)
- [IndicMedDialog: A Parallel Multi-Turn Medical Dialogue Dataset for Accessible Healthcare in Indic Languages](https://mlsanity.com/paper/2605.13292/) (2026)

## Abstract (arXiv)

Dialogue systems for Automatic Differential Diagnosis (ADD) have a wide range of real-life applications. These dialogue systems are promising for providing easy access and reducing medical costs. Building end-to-end ADD dialogue systems requires dialogue training datasets. However, to the best of our knowledge, there is no publicly available ADD dialogue dataset in English (although non-English datasets exist). Driven by this, we introduce MDDial, the first differential diagnosis dialogue dataset in English which can aid to build and evaluate end-to-end ADD dialogue systems. Additionally, earlier studies present the accuracy of diagnosis and symptoms either individually or as a combined weighted score. This method overlooks the connection between the symptoms and the diagnosis. We introduce a unified score for the ADD system that takes into account the interplay between symptoms and diagnosis. This score also indicates the system's reliability. To the end, we train two moderate-size of language models on MDDial. Our experiments suggest that while these language models can perform well on many natural language understanding tasks, including dialogue tasks in the general domain, they struggle to relate relevant symptoms and disease and thus have poor performance on MDDial. MDDial will be released publicly to aid the study of ADD dialogue research.