# Transduce and Speak: Neural Transducer for Text-to-Speech with Semantic Token Prediction

arXiv: https://arxiv.org/abs/2311.02898 · HTML: https://mlsanity.com/paper/2311.02898/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 23 references that are in the map.
- First posted on arXiv 6 Nov 2023.

## Builds on (most cited first, 18 of 18)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech](https://mlsanity.com/paper/2105.06337/) (2021)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search](https://mlsanity.com/paper/2005.11129/) (2020)
- [Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision](https://mlsanity.com/paper/2302.03540/) (2023)
- [Pruned RNN-T for fast, memory-efficient ASR training](https://mlsanity.com/paper/2206.13236/) (2022)
- [Transfer Learning Framework for Low-Resource Text-to-Speech using a Large-Scale Unlabeled Speech Corpus](https://mlsanity.com/paper/2203.15447/) (2022)
- [Visually Guided Self Supervised Learning of Speech Representations](https://mlsanity.com/paper/2001.04316/) (2020)
- [Initial investigation of an encoder-decoder end-to-end TTS framework using marginalization of monotonic hard latent alignments](https://mlsanity.com/paper/1908.11535/) (2019)

## Built on by (most cited first, 7 of 7)

- [SpeakStream: Streaming Text-to-Speech with Interleaved Data](https://mlsanity.com/paper/2505.19206/) (2025)
- [Zero-Shot Streaming Text to Speech Synthesis with Transducer and Auto-Regressive Modeling](https://mlsanity.com/paper/2505.19669/) (2025)
- [Utilizing Neural Transducers for Two-Stage Text-to-Speech via Semantic Token Prediction](https://mlsanity.com/paper/2401.01498/) (2024)
- [Transducer Consistency Regularization for Speech to Text Applications](https://mlsanity.com/paper/2410.07491/) (2024)
- [TTS-Transducer: End-to-End Speech Synthesis with Neural Transducer](https://mlsanity.com/paper/2501.06320/) (2025)
- [Prosodic Boundary-Aware Streaming Generation for LLM-Based TTS with Streaming Text Input](https://mlsanity.com/paper/2603.06444/) (2026)
- [SegINR: Segment-wise Implicit Neural Representation for Sequence Alignment in Neural Text-to-Speech](https://mlsanity.com/paper/2410.04690/) (2024)

## Abstract (arXiv)

We introduce a text-to-speech(TTS) framework based on a neural transducer. We use discretized semantic tokens acquired from wav2vec2.0 embeddings, which makes it easy to adopt a neural transducer for the TTS framework enjoying its monotonic alignment constraints. The proposed model first generates aligned semantic tokens using the neural transducer, then synthesizes a speech sample from the semantic tokens using a non-autoregressive(NAR) speech generator. This decoupled framework alleviates the training complexity of TTS and allows each stage to focus on 1) linguistic and alignment modeling and 2) fine-grained acoustic modeling, respectively. Experimental results on the zero-shot adaptive TTS show that the proposed model exceeds the baselines in speech quality and speaker similarity via objective and subjective measures. We also investigate the inference speed and prosody controllability of our proposed model, showing the potential of the neural transducer for TTS frameworks.