SEmoEdit: Probing and Harnessing the Editability of Pre-trained Speech Flows
On the map: Diffusion models → Audio, speech and music
Abstract (arXiv)
Existing training-based speech emotion editing methods often require substantial task-specific training and can be unstable. This motivates us to investigate whether the pretrained generative dynamics of large-scale text-to-speech (TTS) models can be directly manipulated for training-free emotion editing. To answer this question, we probe the editability of pretrained flow-matching and hybrid TTS models by constructing a controlled test set and systematically diagnosing editing effects along the generative trajectory. Our analysis reveals that pretrained TTS models are substantially editable in emotion, but such editability is architecture- and trajectory-dependent and can be disrupted by early flow-matching steps, while cross-speaker emotion transport carries additional acoustic attributes beyond emotion. To address these limitations, we propose SEmoEdit, the first training-free framework that formulates emotion editing as dynamic velocity transport between source and target emotions, enabling robust, flow-based speech emotion editing directly within pretrained TTS models. SEmoEdit unifies three core operations: emotion replacement, emotion erasure, and continuous emotion interpolation, requiring neither parameter updates nor task-specific optimization. To systematically evaluate these capabilities, we introduce SEmoEditBench, a dataset comprising 600 editing cases, and conduct extensive experiments across state-of-the-art (SOTA) models and backbones. Our results show that SEmoEdit is highly effective and broadly applicable, outperforming existing training-based and activation-steering methods. Ultimately, this work reveals that pretrained speech flows possess rich, latent emotion-editing capabilities, providing useful guidance for real applications. Code, benchmark, and Audio samples are available at https://github.com/imxtx/SEmoEdit.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 32 of its 38 references that are in the map. First posted on arXiv 28 Sep 2026.
Builds on
32 of 38 references.
Closest ancestors · 5
Same direction, newest first
- 2026VoiceDesigner: Text-to-Voice Generation and Editing via Unified Diffusion Modeling and Data Augmentation12 Aug
- 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models13 Dec
- 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching9 Oct
- 2023VoiceFlow: Efficient Text-to-Speech with Rectified Flow Matching10 Sep
- 2023Matcha-TTS: A fast TTS architecture with conditional flow matching6 Sep
Other references · 27
- 2022Flow Matching for Generative Modelingcited by 5,426
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2024FlowEdit: Inversion-Free Text-Based Editing Using Pre-Trained Flow Modelscited by 214
- 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 196
- 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 149
- 2021Emotional Voice Conversion: Theory, Databases and ESDcited by 113
- 2022PromptTTS: Controllable Text-to-Speech with Text Descriptionscited by 106
- 2026Qwen3-TTS Technical Reportcited by 106
- 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 85
- 2024The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speechcited by 78
- 2025EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Promptingcited by 33
- 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 22
15 more
- 2024EmoSphere++: Emotion-Controllable Zero-Shot Text-to-Speech via Emotion-Adaptive Spherical Vectorcited by 19
- 2025EmoSteer-TTS: Fine-Grained and Training-Free Emotion-Controllable Text-to-Speech via Activation Steeringcited by 18
- 2024ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Controlcited by 17
- 2025Ming-UniAudio: Speech LLM for Joint Understanding, Generation and Editing with Unified Representationcited by 15
- 2025Word-Level Emotional Expression Control in Zero-Shot Text-to-Speech Synthesiscited by 7
- 2026CoCoEmo: Composable and Controllable Human-Like Emotional TTS via Activation Steeringcited by 7
- 2025Step-Audio-EditX Technical Reportcited by 6
- 2026EmoShift: Lightweight Activation Steering for Enhanced Emotion-Aware Speech Synthesiscited by 5
- 2026MMAE: A Massive Multitask Audio Editing Benchmarkcited by 5
- 2026SpeechEditBench: A Bilingual Multi-Attribute Benchmark for Instruction-Guided Speech Editingcited by 4
- 2022Emotion Selectable End-to-End Text-based Speech Editingcited by 3
- 2026A Unified Neural Codec Language Model for Selective Editable Text to Speech Generationcited by 2
- 2026Sparse Autoencoders for Interpretable Emotion Control in Text-to-Speechcited by 2
- 2026Bagpiper-Edit: Zero-Shot Open-Ended Audio Editing via Rich-Captioncited by 1
- 2026AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editingcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025TTS-CtrlNet: Time varying emotion aligned text-to-speech generation with ControlNet
- 2025Towards Emotionally Consistent Text-Based Speech Editing: Introducing EmoCorrector and The ECD-TSE Dataset
- 2024Hierarchical Control of Emotion Rendering in Speech Synthesis
- 2026DUET: Unified Dual-Space Emotion Control for Diffusion and Flow-Matching Driven Text-to-Speech
- 2024Enhancing Emotional Text-to-Speech Controllability with Natural Language Guidance through Contrastive Learning and Diffusion Models
- 2026TRACE-EVC: Text-Guided Relative Affective Control for Zero-Shot Emotional Voice Conversion ↗
- 2025Emotion-Aligned Generation in Diffusion Text to Speech Models via Preference-Guided Optimization
- 2026Edit Content, Preserve Acoustics: Imperceptible Text-Based Speech Editing via Self-Consistency Rewards ↗