# Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis

arXiv: https://arxiv.org/abs/2411.01156 · HTML: https://mlsanity.com/paper/2411.01156/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Nov 2024.

## Built on by (most cited first, 30 of 48)

- [Qwen3-TTS Technical Report](https://mlsanity.com/paper/2601.15621/) (2026)
- [IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech](https://mlsanity.com/paper/2506.21619/) (2025)
- [Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation](https://mlsanity.com/paper/2510.01284/) (2025)
- [UniVerse-1: Unified Audio-Video Generation via Stitching of Experts](https://mlsanity.com/paper/2509.06155/) (2025)
- [Recent Advances in Discrete Speech Tokens: A Review](https://mlsanity.com/paper/2502.06490/) (2025)
- [IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System](https://mlsanity.com/paper/2502.05512/) (2025)
- [OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Models](https://mlsanity.com/paper/2604.00688/) (2026)
- [Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision](https://mlsanity.com/paper/2503.01879/) (2025)
- [TouchTTS: An Embarrassingly Simple TTS Framework that Everyone Can Touch](https://mlsanity.com/paper/2412.08237/) (2024)
- [Fish Audio S2 Technical Report](https://mlsanity.com/paper/2603.08823/) (2026)
- [TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systems](https://mlsanity.com/paper/2506.19441/) (2025)
- [DarkStream: real-time speech anonymization with low latency](https://mlsanity.com/paper/2509.04667/) (2025)
- [OmniGuard: Unified Omni-Modal Guardrails with Deliberate Reasoning](https://mlsanity.com/paper/2512.02306/) (2025)
- [Stream-Voice-Anon: Enhancing Utility of Real-Time Speaker Anonymization via Neural Audio Codec and Language Models](https://mlsanity.com/paper/2601.13948/) (2026)
- [SafeSpeech: Robust and Universal Voice Protection Against Malicious Speech Synthesis](https://mlsanity.com/paper/2504.09839/) (2025)
- [Accelerating Autoregressive Speech Synthesis Inference With Speech Speculative Decoding](https://mlsanity.com/paper/2505.15380/) (2025)
- [IndexTTS 2.5 Technical Report](https://mlsanity.com/paper/2601.03888/) (2026)
- [ArVoice: A Multi-Speaker Dataset for Arabic Speech Synthesis](https://mlsanity.com/paper/2505.20506/) (2025)
- [MIKU-PAL: An Automated and Standardized Multi-Modal Method for Speech Paralinguistic and Affect Labeling](https://mlsanity.com/paper/2505.15772/) (2025)
- [Shallow Flow Matching for Coarse-to-Fine Text-to-Speech Synthesis](https://mlsanity.com/paper/2505.12226/) (2025)
- [UniTTS: An end-to-end TTS system without decoupling of acoustic and semantic information](https://mlsanity.com/paper/2505.17426/) (2025)
- [A Survey of Threats Against Voice Authentication and Anti-Spoofing Systems](https://mlsanity.com/paper/2508.16843/) (2025)
- [Position: It's Time to Act on the Risk of Efficient Personalized Text Generation](https://mlsanity.com/paper/2502.06560/) (2025)
- [Transcript-Prompted Whisper with Dictionary-Enhanced Decoding for Japanese Speech Annotation](https://mlsanity.com/paper/2506.07646/) (2025)
- [Spectral Masking and Interpolation Attack (SMIA): A Black-box Adversarial Attack against Voice Authentication and Anti-Spoofing Systems](https://mlsanity.com/paper/2509.07677/) (2025)
- [FlashTTS: Fast Streaming TTS with MTP Acceleration and X-pred Mean Flow Distillation](https://mlsanity.com/paper/2606.09141/) (2026)
- [An Evaluation Framework for Text-to-Speech Voice Reconstruction](https://mlsanity.com/paper/2606.21343/) (2026)
- [TontaubeV1: Streaming Text-to-Speech with Hierarchical Codec Modeling and Bounded Context](https://mlsanity.com/paper/2609.08703/) (2026)
- [RWKVTTS: Yet another TTS based on RWKV-7](https://mlsanity.com/paper/2504.03289/) (2025)
- [WaveLLDM: Design and Development of a Lightweight Latent Diffusion Model for Speech Enhancement and Restoration](https://mlsanity.com/paper/2508.21153/) (2025)

## Abstract (arXiv)

Text-to-Speech (TTS) systems face ongoing challenges in processing complex linguistic features, handling polyphonic expressions, and producing natural-sounding multilingual speech - capabilities that are crucial for future AI applications. In this paper, we present Fish-Speech, a novel framework that implements a serial fast-slow Dual Autoregressive (Dual-AR) architecture to enhance the stability of Grouped Finite Scalar Vector Quantization (GFSQ) in sequence generation tasks. This architecture improves codebook processing efficiency while maintaining high-fidelity outputs, making it particularly effective for AI interactions and voice cloning. Fish-Speech leverages Large Language Models (LLMs) for linguistic feature extraction, eliminating the need for traditional grapheme-to-phoneme (G2P) conversion and thereby streamlining the synthesis pipeline and enhancing multilingual support. Additionally, we developed FF-GAN through GFSQ to achieve superior compression ratios and near 100% codebook utilization. Our approach addresses key limitations of current TTS systems while providing a foundation for more sophisticated, context-aware speech synthesis. Experimental results show that Fish-Speech significantly outperforms baseline models in handling complex linguistic scenarios and voice cloning tasks, demonstrating its potential to advance TTS technology in AI applications. The implementation is open source at hrefhttps://github.com/fishaudio/fish-speechhttps://github.com/fishaudio/fish-speech.