# ToneUnit: A Speech Discretization Approach for Tonal Language Speech Synthesis

arXiv: https://arxiv.org/abs/2406.08989 · HTML: https://mlsanity.com/paper/2406.08989/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jun 2024.

## Built on by (most cited first, 6 of 6)

- [Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens](https://mlsanity.com/paper/2503.01710/) (2025)
- [EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions](https://mlsanity.com/paper/2409.18042/) (2024)
- [Do Discrete Self-Supervised Representations of Speech Capture Tone Distinctions?](https://mlsanity.com/paper/2410.19935/) (2024)
- [How Far Do SSL Speech Models Listen for Tone? Temporal Focus of Tone Representation under Low-resource Transfer](https://mlsanity.com/paper/2511.12285/) (2025)
- [DSA-Tokenizer: Disentangled Semantic-Acoustic Tokenization via Flow Matching-based Hierarchical Fusion](https://mlsanity.com/paper/2601.09239/) (2026)
- [Lexical Tone is Hard to Quantize: Probing Discrete Speech Units in Mandarin and Yor`ub'a](https://mlsanity.com/paper/2604.07467/) (2026)

## Abstract (arXiv)

Representing speech as discretized units has numerous benefits in supporting downstream spoken language processing tasks. However, the approach has been less explored in speech synthesis of tonal languages like Mandarin Chinese. Our preliminary experiments on Chinese speech synthesis reveal the issue of "tone shift", where a synthesized speech utterance contains correct base syllables but incorrect tones. To address the issue, we propose the ToneUnit framework, which leverages annotated data with tone labels as CTC supervision to learn tone-aware discrete speech units for Mandarin Chinese speech. Our findings indicate that the discrete units acquired through the TonUnit resolve the "tone shift" issue in synthesized Chinese speech and yield favorable results in English synthesis. Moreover, the experimental results suggest that finite scalar quantization enhances the effectiveness of ToneUnit. Notably, ToneUnit can work effectively even with minimal annotated data.