# Spoken DialogSum: An Emotion-Rich Conversational Dataset for Spoken Dialogue Summarization

arXiv: https://arxiv.org/abs/2512.14687 · HTML: https://mlsanity.com/paper/2512.14687/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 68 references that are in the map.
- First posted on arXiv 16 Dec 2025.

## Builds on (most cited first, 30 of 30)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization](https://mlsanity.com/paper/1911.12237/) (2019)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio](https://mlsanity.com/paper/2106.06909/) (2021)
- [DialogSum: A Real-Life Scenario Dialogue Summarization Dataset](https://mlsanity.com/paper/2105.06762/) (2021)
- [DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehension](https://mlsanity.com/paper/1902.00164/) (2019)
- [How2: A Large-scale Dataset for Multimodal Language Understanding](https://mlsanity.com/paper/1811.00347/) (2018)
- [MuTual: A Dataset for Multi-Turn Dialogue Reasoning](https://mlsanity.com/paper/2004.04494/) (2020)
- [Yara Parser: A Fast and Accurate Dependency Parser](https://mlsanity.com/paper/1503.06733/) (2015)
- [MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarization](https://mlsanity.com/paper/2103.06410/) (2021)
- [SummScreen: A Dataset for Abstractive Screenplay Summarization](https://mlsanity.com/paper/2104.07091/) (2021)
- [Multi-View Sequence-to-Sequence Models with Conversational Structure for Abstractive Dialogue Summarization](https://mlsanity.com/paper/2010.01672/) (2020)
- [DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speech](https://mlsanity.com/paper/2207.01063/) (2022)
- [ConvoSumm: Conversation Summarization Benchmark and Improved Abstractive Summarization with Argument Mining](https://mlsanity.com/paper/2106.00829/) (2021)
- [Structure-Aware Abstractive Conversation Summarization via Discourse and Action Graphs](https://mlsanity.com/paper/2104.08400/) (2021)
- [DiaSynth: Synthetic Dialogue Generation Framework for Low Resource Dialogue Applications](https://mlsanity.com/paper/2409.19020/) (2024)
- [Speech-based Age and Gender Prediction with Transformers](https://mlsanity.com/paper/2306.16962/) (2023)
- [Yeah, Right, Uh-Huh: A Deep Learning Backchannel Predictor](https://mlsanity.com/paper/1706.01340/) (2017)
- [CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech](https://mlsanity.com/paper/2506.02863/) (2025)
- [SponTTS: modeling and transferring spontaneous style for TTS](https://mlsanity.com/paper/2311.07179/) (2023)
- [Semi-Supervised Dialogue Abstractive Summarization via High-Quality Pseudolabel Selection](https://mlsanity.com/paper/2403.04073/) (2024)
- [Mutual Reinforcement of LLM Dialogue Synthesis and Summarization Capabilities for Few-Shot Dialogue Summarization](https://mlsanity.com/paper/2502.17328/) (2025)
- [Paired by the Teacher: Turning Unpaired Data into High-Fidelity Pairs for Low-Resource Text Generation](https://mlsanity.com/paper/2509.25144/) (2025)

## Built on by (most cited first, 2 of 2)

- [End-to-End Voice Intent Recognition for Spontaneous Human-Drone Interaction with Naive Users](https://mlsanity.com/paper/2606.24910/) (2026)
- [CARES: A Controlled Synthetic Benchmark of Speaker Reactions to Sound](https://mlsanity.com/paper/2610.10208/) (2026)

## Abstract (arXiv)

Recent audio language models can follow long conversations. However, research on emotion-aware or spoken dialogue summarization is constrained by the lack of data that links speech, summaries, and paralinguistic cues. We introduce Spoken DialogSum, the first corpus aligning raw conversational audio with factual summaries, emotion-rich summaries, and utterance-level labels for speaker age, gender, and emotion. The dataset is built in two stages: first, an LLM rewrites DialogSum scripts with Switchboard-style fillers and back-channels, then tags each utterance with emotion, pitch, and speaking rate. Second, an expressive TTS engine synthesizes speech from the tagged scripts, aligned with paralinguistic labels. Spoken DialogSum comprises 13,460 emotion-diverse dialogues, each paired with both a factual and an emotion-focused summary. We release an online demo at https://fatfat-emosum.github.io/EmoDialog-Sum-Audio-Samples/, with plans to release the full dataset in the near future. Baselines show that an Audio-LLM raises emotional-summary ROUGE-L by 28% relative to a cascaded ASR-LLM system, confirming the value of end-to-end speech modeling.