mlsanityData to 14 Oct 2026

PilotTTS: A Disciplined Modular Recipe for Competitive Speech Synthesis

arXiv 2605.27258 ↗First posted 26 May 2026cs.SD cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Building state-of-the-art text-to-speech (TTS) systems typically demands millions of hours of proprietary data and complex multi-stage architectures, creating substantial barriers for resource-constrained research teams. In this report, we present PilotTTS, a lightweight autoregressive TTS system that achieves competitive performance through minimalist architecture and rigorous data engineering. PilotTTS is trained on only 200K hours of data processed entirely with open-source tools. Specifically, our contributions are: (1) a reproducible multi-stage data processing pipeline covering quality assessment, label annotation, and filtering, and (2) a compact model architecture that employs Q-Former-based conditioning to decouple speaker identity from speaking style via cross-sample paired training. Within a unified framework, PilotTTS supports zero-shot voice cloning, emotion synthesis (11 categories), paralinguistic synthesis (4 categories), and Chinese dialect synthesis (14 dialects). On the Seed-TTS Eval benchmark, PilotTTS achieves the lowest WER of 1.50% on test-en, a CER of 0.87% on test-zh, and the highest speaker similarity on both test sets (0.862 and 0.815), outperforming systems trained on significantly larger datasets. We release the complete data pipeline recipe, pretrained weights, and code at https://github.com/AMAPVOICE/PilotTTS.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 32 references that are in the map. First posted on arXiv 26 May 2026.

Builds on

31 of 32 references.

References · 31

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2022Scalable Diffusion Models with Transformerscited by 6,054
  4. 2022Flow Matching for Generative Modelingcited by 5,702
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  7. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  8. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 361
  9. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  10. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  11. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
  12. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
19 more
  1. 2020DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressorscited by 241
  2. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 235
  3. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 226
  4. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 150
  5. 2022Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognitioncited by 144
  6. 2026Qwen3-TTS Technical Reportcited by 118
  7. 2026Qwen3-ASR Technical Reportcited by 118
  8. 2023Powerset multi-class cross entropy loss for neural speaker diarizationcited by 111
  9. 2023CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Maskingcited by 102
  10. 2025DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generationcited by 57
  11. 2025MiniMax-Speech: Intrinsic Zero-Shot Text-to-Speech with a Learnable Speaker Encodercited by 48
  12. 2025FireRedASR: Open-Source Industrial-Grade Mandarin Speech Recognition Models from Encoder-Decoder to LLM Integrationcited by 46
  13. 2025VoxCPM: Tokenizer-Free TTS for Context-Aware Speech Generation and True-to-Life Voice Cloningcited by 42
  14. 2025FireRedTTS-2: Towards Long Conversational Speech Generation for Podcast and Chatbotcited by 35
  15. 2025VibeVoice Technical Reportcited by 29
  16. 2025IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech Systemcited by 28
  17. 20243D-Speaker-Toolkit: An Open-Source Toolkit for Multimodal Speaker Verification and Diarizationcited by 16
  18. 2024Towards Efficient Visual-Language Alignment of the Q-Former for Visual Reasoning Taskscited by 9
  19. 2026Fish Audio S2 Technical Reportcited by 9

Built on by

Not cited yet.