# Semantic-VAE: Semantic-Alignment Latent Representation for Better Speech Synthesis

arXiv: https://arxiv.org/abs/2509.22167 · HTML: https://mlsanity.com/paper/2509.22167/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Sep 2025.

## Built on by (most cited first, 12 of 12)

- [dots.tts Technical Report](https://mlsanity.com/paper/2606.07080/) (2026)
- [LongCat-AudioDiT: High-Fidelity Diffusion Text-to-Speech in the Waveform Latent Space](https://mlsanity.com/paper/2603.29339/) (2026)
- [WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling](https://mlsanity.com/paper/2605.06407/) (2026)
- [MoLingo: Motion-Language Alignment for Text-to-Human Motion Generation](https://mlsanity.com/paper/2512.13840/) (2025)
- [WavTTS: Towards High-Quality Zero-Shot TTS via Direct Raw Waveform Modeling](https://mlsanity.com/paper/2606.03455/) (2026)
- [GROW: Group-Relative Advantage-Weighted On-Policy Reinforcement Learning of Autoregressive-Diffusion Text-to-Speech model](https://mlsanity.com/paper/2608.03215/) (2026)
- [Representation-Regularized Convolutional Audio Transformer for Audio Understanding](https://mlsanity.com/paper/2601.21612/) (2026)
- [S2Accompanist: A Semantic-Aware and Structure-Guided Diffusion Model for Music Accompaniment Generation](https://mlsanity.com/paper/2605.17414/) (2026)
- [LoSATok: Low-Dimensional Semantic-Acoustic Tokenizer for Cross-Domain Audio Understanding and Generation](https://mlsanity.com/paper/2605.27840/) (2026)
- [HoliDubber: Holistic Video Dubbing for Complex Acoustic Scenes via Text-Guided Audio Synthesis](https://mlsanity.com/paper/2606.09098/) (2026)
- [CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents](https://mlsanity.com/paper/2608.08638/) (2026)
- [KABURI-TTS: Phoneme-Keyed Activity-conditioned Bi-channel Utterance Rendering for Interaction](https://mlsanity.com/paper/2609.07200/) (2026)

## Abstract (arXiv)

Mel-spectrograms have been widely used in zero-shot text-to-speech (TTS); their inherent redundancy leads to inefficiency in text-speech alignment. Compact VAE-based latent representations have emerged as a stronger alternative but exhibit an optimization dilemma: higher-dimensional latents improve reconstruction quality and speaker similarity but degrade intelligibility, while lower-dimensional latents improve intelligibility at the cost of reconstruction fidelity. To overcome this dilemma, we propose Semantic-VAE, which uses semantic alignment regularization in the latent space. This design alleviates the reconstruction-generation trade-off by capturing semantic structure in high-dimensional latent representations. When integrated into F5-TTS, our method achieves 2.10% WER and 0.64 speaker similarity on LibriSpeech-PC, outperforming mel-based systems and vanilla acoustic VAE baselines with improved training efficiency. Demo and codes: https://zhikangniu.github.io/semantic-vae/