mlsanityData to 14 Oct 2026

NVAlign: Direct-Gradient Optimization for Non-Verbal Control in Continuous Autoregressive Flow Matching Text-to-Speech

arXiv 2609.31892 ↗First posted 25 Sep 2026cs.SD cs.AI cs.CLcited by 0

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

While modern text-to-speech (TTS) systems generate highly natural speech and support inline non-verbal vocalization (NVV) tags, accurate control over these events remains challenging. A key gap is the lack of established post-training methods for non-verbal control in continuous autoregressive flow-matching TTS. To this end, we present NVAlign, a direct-gradient post-training framework for NVV tag-following in this architecture. We first perform supervised fine-tuning (SFT) of TTS models and an NVV-aware automatic speech recognition (NV-ASR) model on NVV-annotated speech, then freeze the NV-ASR model to serve as the reward model for post-training. A two-step gradient surrogate enables efficient reward backpropagation through the flow-matching sampler to jointly update the autoregressive backbone and acoustic flow head. Fidelity penalties and reference-velocity regularization help preserve speaker similarity and speech quality. Results from NVV-SuperBench and human listening evaluations show that NVAlign improves tag-following accuracy over SFT and Flow-GRPO baselines. These findings demonstrate that direct reward-gradient optimization can improve non-verbal control in continuous autoregressive flow-matching TTS. Audio samples are available at https://nvalign.github.io/.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 37 of its 45 references that are in the map. First posted on arXiv 25 Sep 2026.

Builds on

37 of 45 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026GROW: Group-Relative Advantage-Weighted On-Policy Reinforcement Learning of Autoregressive-Diffusion Text-to-Speech model4 Aug
  2. 2026FlowTTS-GRPO: Online Reinforcement Learning with Multi-Objective Reward Optimization for Flow-Matching Based Text-to-Speech22 Jun
  3. 2025Direct Preference Optimization for Speech Autoregressive Diffusion Models23 Sep
  4. 2025DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation6 Feb
  5. 2024DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesis14 Oct

Other references · 32

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 661
  4. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 628
  5. 2025Qwen3-Omni Technical Reportcited by 497
  6. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 448
  7. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
  8. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 288
  9. 2022Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognitioncited by 144
  10. 2024Deep Reward Supervisions for Tuning Text-to-Image Diffusion Modelscited by 59
  11. 2025VoxCPM: Tokenizer-Free TTS for Context-Aware Speech Generation and True-to-Life Voice Cloningcited by 42
  12. 2025From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Onescited by 42
20 more
  1. 2026Flow-OPD: On-Policy Distillation for Flow Matching Modelscited by 34
  2. 2025AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluationcited by 21
  3. 2026VoxCPM2 Technical Reportcited by 17
  4. 2025Differentiable Reward Optimization for LLM based TTS systemcited by 15
  5. 2025NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speechcited by 14
  6. 2024Effective Pre-Training of Audio Transformers for Sound Event Detectioncited by 12
  7. 2024Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Likecited by 11
  8. 2025A Scalable Pipeline for Enabling Non-Verbal Speech Generation and Understandingcited by 11
  9. 2026Fish Audio S2 Technical Reportcited by 9
  10. 2025Group Relative Policy Optimization for Text-to-Speech with Large Language Modelscited by 7
  11. 2025SynParaSpeech: Automated Synthesis of Paralinguistic Datasets for Speech Generation and Understandingcited by 6
  12. 2025MNV-17: A High-Quality Performative Mandarin Dataset for Nonverbal Vocalization Recognition in Speechcited by 6
  13. 2026IndexTTS 2.5 Technical Reportcited by 6
  14. 2026NV-Bench: Benchmark of Nonverbal Vocalization Synthesis for Expressive Text-to-Speech Generationcited by 6
  15. 2026LeapAlign: Post-Training Flow Matching Models at Any Generation Step by Building Two-Step Trajectoriescited by 6
  16. 2025TTS-1 Technical Reportcited by 3
  17. 2026Preference Optimization for Non-Verbal Vocalization Synthesiscited by 2
  18. 2026Preference Optimization with LALM Feedback for Continuous Autoregressive Non-Verbal Vocalization Generationcited by 2
  19. 2026NVV-SuperBench: Beyond Words, Beyond Quality-Benchmarking Nonverbal Vocalizations in Speech Generationcited by 1
  20. 2026NVMOS: Non-Verbal Vocalization Quality Assessment in Speechcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Models
  2. 2026CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents
  3. 2026Mask, Sample, Revise: A Revisable CTMC Inference Stack for Guided Discrete Flow Matching Text-to-Speech
  4. 2026ARCHI-TTS: A flow-matching-based Text-to-Speech Model with Self-supervised Semantic Aligner and Accelerated Inference
  5. 2026Speaker Identity in Non-Verbal Vocalizations: Conditional Distillation and Mixture of Experts Approach
  6. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching
  7. 2026End-to-End Training for Discrete Token LLM based TTS System
  8. 2026ReaFlow-TTS: Realization-Conditioned Flow Matching for High-Quality and Controllable Speech Synthesis