# AttentionStitch: How Attention Solves the Speech Editing Problem

arXiv: https://arxiv.org/abs/2403.04804 · HTML: https://mlsanity.com/paper/2403.04804/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 15 references that are in the map.
- First posted on arXiv 5 Mar 2024.

## Builds on (most cited first, 7 of 7)

- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing](https://mlsanity.com/paper/2203.09690/) (2022)
- [EditSpeech: A Text Based Speech Editing System Using Partial Inference and Bidirectional Fusion](https://mlsanity.com/paper/2107.01554/) (2021)
- [EdiTTS: Score-based Editing for Controllable Text-to-Speech](https://mlsanity.com/paper/2110.02584/) (2021)
- [SpeechPainter: Text-conditioned Speech Inpainting](https://mlsanity.com/paper/2202.07273/) (2022)
- [Context-Aware Prosody Correction for Text-Based Speech Editing](https://mlsanity.com/paper/2102.08328/) (2021)
- [MaskedSpeech: Context-aware Speech Synthesis with Masking Strategy](https://mlsanity.com/paper/2211.06170/) (2022)

## Built on by (most cited first, 1 of 1)

- [FluentEditor2: Text-based Speech Editing by Modeling Multi-Scale Acoustic and Prosody Consistency](https://mlsanity.com/paper/2410.03719/) (2024)

## Abstract (arXiv)

The generation of natural and high-quality speech from text is a challenging problem in the field of natural language processing. In addition to speech generation, speech editing is also a crucial task, which requires the seamless and unnoticeable integration of edited speech into synthesized speech. We propose a novel approach to speech editing by leveraging a pre-trained text-to-speech (TTS) model, such as FastSpeech 2, and incorporating a double attention block network on top of it to automatically merge the synthesized mel-spectrogram with the mel-spectrogram of the edited text. We refer to this model as AttentionStitch, as it harnesses attention to stitch audio samples together. We evaluate the proposed AttentionStitch model against state-of-the-art baselines on both single and multi-speaker datasets, namely LJSpeech and VCTK. We demonstrate its superior performance through an objective and a subjective evaluation test involving 15 human participants. AttentionStitch is capable of producing high-quality speech, even for words not seen during training, while operating automatically without the need for human intervention. Moreover, AttentionStitch is fast during both training and inference and is able to generate human-sounding edited speech.