# VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speech

arXiv: https://arxiv.org/abs/2401.14321 · HTML: https://mlsanity.com/paper/2401.14321/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jan 2024.

## Built on by (most cited first, 23 of 23)

- [F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching](https://mlsanity.com/paper/2410.06885/) (2024)
- [CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models](https://mlsanity.com/paper/2412.10117/) (2024)
- [CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training](https://mlsanity.com/paper/2505.17589/) (2025)
- [VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild](https://mlsanity.com/paper/2403.16973/) (2024)
- [Recent Advances in Discrete Speech Tokens: A Review](https://mlsanity.com/paper/2502.06490/) (2025)
- [VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment](https://mlsanity.com/paper/2406.07855/) (2024)
- [RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesis](https://mlsanity.com/paper/2404.03204/) (2024)
- [Fast and High-Quality Auto-Regressive Speech Synthesis via Speculative Decoding](https://mlsanity.com/paper/2410.21951/) (2024)
- [Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decoding](https://mlsanity.com/paper/2410.13839/) (2024)
- [Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesis](https://mlsanity.com/paper/2504.10352/) (2025)
- [SpeakStream: Streaming Text-to-Speech with Interleaved Data](https://mlsanity.com/paper/2505.19206/) (2025)
- [DMOSpeech 2: Reinforcement Learning for Duration Prediction in Metric-Optimized Speech Synthesis](https://mlsanity.com/paper/2507.14988/) (2025)
- [AniME: Adaptive Multi-Agent Planning for Long Animation Generation](https://mlsanity.com/paper/2508.18781/) (2025)
- [Robust and Unbounded Length Generalization in Autoregressive Transformer-Based Text-to-Speech](https://mlsanity.com/paper/2410.22179/) (2024)
- [Voice Cloning: Comprehensive Survey](https://mlsanity.com/paper/2505.00579/) (2025)
- [Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignment](https://mlsanity.com/paper/2505.19595/) (2025)
- [Robust and Efficient Autoregressive Speech Synthesis with Dynamic Chunk-wise Prediction Policy](https://mlsanity.com/paper/2506.22023/) (2025)
- [OmniCustom: Sync Audio-Video Customization Via Joint Audio-Video Generation Model](https://mlsanity.com/paper/2602.12304/) (2026)
- [TTS-Transducer: End-to-End Speech Synthesis with Neural Transducer](https://mlsanity.com/paper/2501.06320/) (2025)
- [Speech Recognition Model Improves Text-to-Speech Synthesis using Fine-Grained Reward](https://mlsanity.com/paper/2511.17555/) (2025)
- [SegINR: Segment-wise Implicit Neural Representation for Sequence Alignment in Neural Text-to-Speech](https://mlsanity.com/paper/2410.04690/) (2024)
- [Experience-Calibrated Contrastive Decoding for Mitigating Hallucinations in LM-Based Text-to-Speech](https://mlsanity.com/paper/2608.00722/) (2026)
- [Hallucinations in LLMs: A Lifecycle-Based Survey of Causes, Detection, Mitigation, and Prevention](https://mlsanity.com/paper/2608.26168/) (2026)

## Abstract (arXiv)

Recent TTS models with decoder-only Transformer architecture, such as SPEAR-TTS and VALL-E, achieve impressive naturalness and demonstrate the ability for zero-shot adaptation given a speech prompt. However, such decoder-only TTS models lack monotonic alignment constraints, sometimes leading to hallucination issues such as mispronunciation, word skipping and repeating. To address this limitation, we propose VALL-T, a generative Transducer model that introduces shifting relative position embeddings for input phoneme sequence, explicitly indicating the monotonic generation process while maintaining the architecture of decoder-only Transformer. Consequently, VALL-T retains the capability of prompt-based zero-shot adaptation and demonstrates better robustness against hallucinations with a relative reduction of 28.3% in the word error rate.