# FluentEditor: Text-based Speech Editing by Considering Acoustic and Prosody Consistency

arXiv: https://arxiv.org/abs/2309.11725 · HTML: https://mlsanity.com/paper/2309.11725/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 26 references that are in the map.
- First posted on arXiv 21 Sep 2023.

## Builds on (most cited first, 12 of 12)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing](https://mlsanity.com/paper/2203.09690/) (2022)
- [EditSpeech: A Text Based Speech Editing System Using Partial Inference and Bidirectional Fusion](https://mlsanity.com/paper/2107.01554/) (2021)
- [Revisiting Over-Smoothness in Text to Speech](https://mlsanity.com/paper/2202.13066/) (2022)
- [Expressive TTS Training with Frame and Style Reconstruction Loss](https://mlsanity.com/paper/2008.01490/) (2020)
- [EdiTTS: Score-based Editing for Controllable Text-to-Speech](https://mlsanity.com/paper/2110.02584/) (2021)
- [FluentSpeech: Stutter-Oriented Automatic Speech Editing with Context-Aware Diffusion Models](https://mlsanity.com/paper/2305.13612/) (2023)
- [Controllable neural text-to-speech synthesis using intuitive prosodic features](https://mlsanity.com/paper/2009.06775/) (2020)
- [CampNet: Context-Aware Mask Prediction for End-to-End Text-Based Speech Editing](https://mlsanity.com/paper/2202.09950/) (2022)

## Built on by (most cited first, 6 of 6)

- [DiffEditor: Enhancing Speech Editing with Semantic Enrichment and Acoustic Consistency](https://mlsanity.com/paper/2409.12992/) (2024)
- [SeniorTalk: A Chinese Conversation Dataset with Rich Annotations for Super-Aged Seniors](https://mlsanity.com/paper/2503.16578/) (2025)
- [FluentEditor2: Text-based Speech Editing by Modeling Multi-Scale Acoustic and Prosody Consistency](https://mlsanity.com/paper/2410.03719/) (2024)
- [Towards Emotionally Consistent Text-Based Speech Editing: Introducing EmoCorrector and The ECD-TSE Dataset](https://mlsanity.com/paper/2505.20341/) (2025)
- [dots.tts.edit: Precisely Controlled Speech Editing with a Continuous Autoregressive Model](https://mlsanity.com/paper/2608.02673/) (2026)
- [Multi Codec Discrete Diffusion Model for Text Guided Speech Inpainting and Editing](https://mlsanity.com/paper/2608.06424/) (2026)

## Abstract (arXiv)

Text-based speech editing (TSE) techniques are designed to enable users to edit the output audio by modifying the input text transcript instead of the audio itself. Despite much progress in neural network-based TSE techniques, the current techniques have focused on reducing the difference between the generated speech segment and the reference target in the editing region, ignoring its local and global fluency in the context and original utterance. To maintain the speech fluency, we propose a fluency speech editing model, termed textitFluentEditor, by considering fluency-aware training criterion in the TSE training. Specifically, the textitacoustic consistency constraint aims to smooth the transition between the edited region and its neighboring acoustic segments consistent with the ground truth, while the textitprosody consistency constraint seeks to ensure that the prosody attributes within the edited regions remain consistent with the overall style of the original utterance. The subjective and objective experimental results on VCTK demonstrate that our textitFluentEditor outperforms all advanced baselines in terms of naturalness and fluency. The audio samples and code are available at urlhttps://github.com/Ai-S2-Lab/FluentEditor.