# Bridging the Stability-Expressivity Gap: Synthetic Data Scaling and Preference Alignment for Low-Resource Spoken Language Models

arXiv: https://arxiv.org/abs/2605.27383 · HTML: https://mlsanity.com/paper/2605.27383/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 18 of its 67 references that are in the map.
- First posted on arXiv 10 Apr 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Preference alignment for recommenders and RAG](https://mlsanity.com/alignment/preference-alignment-for-recommenders-and-rag/)
- Synthetic data and self-improvement for LLMs / [LLM synthetic data generation](https://mlsanity.com/synth/llm-synthetic-data-generation/)

## Builds on (most cited first, 18 of 18)

- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models](https://mlsanity.com/paper/2412.10117/) (2024)
- [NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models](https://mlsanity.com/paper/2403.03100/) (2024)
- [Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling](https://mlsanity.com/paper/2303.03926/) (2023)
- [SeamlessM4T: Massively Multilingual & Multimodal Machine Translation](https://mlsanity.com/paper/2308.11596/) (2023)
- [PyThaiNLP: Thai Natural Language Processing in Python](https://mlsanity.com/paper/2312.04649/) (2023)
- [Scaling Low-Resource MT via Synthetic Data Generation with LLMs](https://mlsanity.com/paper/2505.14423/) (2025)
- [Typhoon 2: A Family of Open Text and Multimodal Thai Large Language Models](https://mlsanity.com/paper/2412.13702/) (2024)
- [Dolphin: A Large-Scale Automatic Speech Recognition Model for Eastern Languages](https://mlsanity.com/paper/2503.20212/) (2025)
- [Overcoming Data Scarcity in Generative Language Modelling for Low-Resource Languages: A Systematic Review](https://mlsanity.com/paper/2505.04531/) (2025)
- [Multi-Metric Preference Alignment for Generative Speech Restoration](https://mlsanity.com/paper/2508.17229/) (2025)
- [Theoretical Proof that Auto-regressive Language Models Collapse when Real-world Data is a Finite Set](https://mlsanity.com/paper/2412.14872/) (2024)
- [VoiceStar: Robust Zero-Shot Autoregressive TTS with Duration Control and Extrapolation](https://mlsanity.com/paper/2505.19462/) (2025)
- [Direct Preference Optimization for Speech Autoregressive Diffusion Models](https://mlsanity.com/paper/2509.18928/) (2025)
- [Selective Classifier-free Guidance for Zero-shot Text-to-speech](https://mlsanity.com/paper/2509.19668/) (2025)
- [Investigating Stochastic Methods for Prosody Modeling in Speech Synthesis](https://mlsanity.com/paper/2507.00227/) (2025)
- [DisCo-Speech: Controllable Zero-Shot Speech Generation with A Disentangled Speech Codec](https://mlsanity.com/paper/2512.13251/) (2025)
- [UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Models](https://mlsanity.com/paper/2510.04593/) (2025)

## Abstract (arXiv)

Spoken Language Models (SLMs) have emerged as a promising paradigm for speech synthesis by bypassing explicit grapheme-to-phoneme pipelines. However, their effectiveness in low-resource languages remains fundamentally limited by the scarcity of transcribed speech. In practice, synthetic data has become the primary strategy for scaling SLMs in such settings, providing reliable phonetic supervision when real data is insufficient. In this work, we show that this reliance introduces a fundamental trade-off, which we term the Stability-Expressivity Gap: while synthetic data improves phonetic accuracy, it progressively suppresses prosodic variability, ultimately leading to a collapse of expressivity (Synthetic Erosion). To bridge this gap, we propose two self-alignment frameworks. Disentanglement-Guided Self-Alignment (DGSA) recovers expressivity for complex languages by exploiting prosody-timbre separation. For regimes where authentic references are exceptionally limited, Temperature-Driven Self-Critique (TDSC) stabilizes generation through automated exploration and filtering. Our approach outperforms strong commercial systems, including ElevenLabs and Gemini Pro, and enables the first zero-shot voice cloning capability for Lao.