mlsanityData to 14 Oct 2026

FELLE: Autoregressive Speech Synthesis with Token-Wise Coarse-to-Fine Flow Matching

arXiv 2502.11128 ↗First posted 16 Feb 2025cs.CL cs.SD eess.AScited by 8

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

To advance continuous-valued token modeling and temporal-coherence enforcement, we propose FELLE, an autoregressive model that integrates language modeling with token-wise flow matching. By leveraging the autoregressive nature of language models and the generative efficacy of flow matching, FELLE effectively predicts continuous-valued tokens (mel-spectrograms). For each continuous-valued token, FELLE modifies the general prior distribution in flow matching by incorporating information from the previous step, improving coherence and stability. Furthermore, to enhance synthesis quality, FELLE introduces a coarse-to-fine flow-matching mechanism, generating continuous-valued tokens hierarchically, conditioned on the language model's output. Experimental results demonstrate the potential of incorporating flow-matching techniques in autoregressive mel-spectrogram modeling, leading to significant improvements in TTS generation quality, as shown in https://aka.ms/felle.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 36 of its 47 references that are in the map. First posted on arXiv 16 Feb 2025.

Builds on

36 of 47 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Speech Synthesis From Continuous Features Using Per-Token Latent Diffusion21 Oct
  2. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching9 Oct
  3. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models5 Mar
  4. 2024SpeechGPT-Gen: Scaling Chain-of-Information Speech Generation24 Jan
  5. 2023Matcha-TTS: A fast TTS architecture with conditional flow matching6 Sep

Other references · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Classifier-Free Diffusion Guidancecited by 5,847
  4. 2022Flow Matching for Generative Modelingcited by 5,702
  5. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  9. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  10. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  11. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  12. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
19 more
  1. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  2. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  3. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  4. 2017VAE with a VampPriorcited by 248
  5. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 169
  6. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 167
  7. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  8. 2021Generalization Ability of MOS Prediction Networkscited by 69
  9. 2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
  10. 2024ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reorderingcited by 50
  11. 2019MelNet: A Generative Model for Audio in the Frequency Domaincited by 42
  12. 2024VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignmentcited by 38
  13. 2024CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speechcited by 37
  14. 2022Revisiting Over-Smoothness in Text to Speechcited by 32
  15. 2023Explicitly Minimizing the Blur Error of Variational Autoencoderscited by 31
  16. 2024RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesiscited by 26
  17. 2023Discrete Audio Representation as an Alternative to Mel-Spectrograms for Speaker and Speech Recognitioncited by 11
  18. 2023RAMP: Retrieval-Augmented MOS Prediction via Confidence-based Dynamic Weightingcited by 6
  19. 2024Uncertainty-Aware Mean Opinion Score Predictioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 4.

8 papers.

  1. 2025Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesiscited by 10
  2. 2025StreamMel: Real-Time Zero-shot Text-to-Speech via Interleaved Continuous Autoregressive Modelingcited by 3
  3. 2025Next Tokens Denoising for Speech Synthesiscited by 3
  4. 2026Sylber 2.0: A Universal Syllable Embeddingcited by 2
  5. 2025Efficient Speech Language Modeling via Energy Distance in Continuous Latent Spacecited by 1
  6. 2026Investigating Group Relative Policy Optimization for Diffusion Transformer based Text-to-Audio Generationcited by 0
  7. 2026CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latentscited by 0
  8. 2026Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Continuous Autoregressive Modeling with Stochastic Monotonic Alignment for Speech Synthesis
  2. 2025Accelerating Autoregressive Speech Synthesis Inference With Speech Speculative Decoding
  3. 2025Robust and Efficient Autoregressive Speech Synthesis with Dynamic Chunk-wise Prediction Policy
  4. 2025CLEAR: Continuous Latent Autoregressive Modeling for High-quality and Low-latency Speech Synthesis
  5. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesis
  6. 2026ReaFlow-TTS: Realization-Conditioned Flow Matching for High-Quality and Controllable Speech Synthesis
  7. 2026TLDR: Compressing Audio Tokens for Efficient Autoregressive Text-to-Speech
  8. 2024Boosting Diffusion Model for Spectrogram Up-sampling in Text-to-speech: An Empirical Study