# Distributional Metrics for Evaluating Spoken Conversational Systems

arXiv: https://arxiv.org/abs/2609.31719 · HTML: https://mlsanity.com/paper/2609.31719/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 34 references that are in the map.
- First posted on arXiv 21 Sep 2026.

## Builds on (most cited first, 20 of 20)

- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale](https://mlsanity.com/paper/2306.15687/) (2023)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality](https://mlsanity.com/paper/2205.04421/) (2022)
- [Generative Spoken Dialogue Language Modeling](https://mlsanity.com/paper/2203.16502/) (2022)
- [Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities](https://mlsanity.com/paper/2503.04721/) (2025)
- [Universal Phone Recognition with a Multilingual Allophone System](https://mlsanity.com/paper/2002.11800/) (2020)
- [PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models](https://mlsanity.com/paper/2602.06053/) (2026)
- [Full-Duplex-Bench v1.5: Evaluating Overlap Handling for Full-Duplex Speech Models](https://mlsanity.com/paper/2507.23159/) (2025)
- [Evaluating Coherence in Dialogue Systems using Entailment](https://mlsanity.com/paper/1904.03371/) (2019)
- [Full-Duplex-Bench-v2: A Multi-Turn Evaluation Framework for Duplex Dialogue Systems with an Automated Examiner](https://mlsanity.com/paper/2510.07838/) (2025)
- [MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models](https://mlsanity.com/paper/2511.10262/) (2025)
- [Sylber: Syllabic Embedding Representation of Speech from Raw Audio](https://mlsanity.com/paper/2410.07168/) (2024)
- [TTSDS -- Text-to-Speech Distribution Score](https://mlsanity.com/paper/2407.12707/) (2024)
- [TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systems](https://mlsanity.com/paper/2506.19441/) (2025)
- [M3-DuplexBench: A Multi-Turn, Multilingual, Multidomain Benchmark for Full-Duplex Spoken Dialogue Models](https://mlsanity.com/paper/2607.29125/) (2026)
- [Acoustic and Facial Markers of Perceived Conversational Success in Spontaneous Speech](https://mlsanity.com/paper/2604.15322/) (2026)
- [Reference-Based Prosody and Rhythm Evaluation for Spoken Dialogue Systems](https://mlsanity.com/paper/2606.31055/) (2026)
- [SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Models](https://mlsanity.com/paper/2607.05365/) (2026)

## Abstract (arXiv)

Evaluating conversational systems is a difficult and unresolved problem. We introduce the Conversational Distribution Score (CDS), which compares distributions of conversational behaviour using human conversations as a reference. CDS describes speech rate, syllabic rhythm, and turn interaction through eight interpretable features plus a separate two-feature semantic baseline. We compare conversations with two reference scales: one based on conversational success within human dialogue and another contrasting human and synthetic dialogue. Using listener judgments from out-of-domain goal--oriented dialogues, we examine system ranking, preferences between conversations, and ranking stability. Composite CDS recovers five of six listener system comparisons while individual features show strong correlation with listener preferences between conversations. We examine how many minutes and conversations are required before rankings stabilize. These findings support distributional comparisons as a complement to specific interactional metrics to evaluate conversations and conversational models while showing their interpretable value.