mlsanityData to 14 Oct 2026

A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing

arXiv 2203.09690 ↗First posted 18 Mar 2022eess.AS cs.CL cs.SDcited by 40

Not on the map. Why?

Abstract (arXiv)

Recently, speech representation learning has improved many speech-related tasks such as speech recognition, speech classification, and speech-to-text translation. However, all the above tasks are in the direction of speech understanding, but for the inverse direction, speech synthesis, the potential of representation learning is yet to be realized, due to the challenging nature of generating high-quality speech. To address this problem, we propose our framework, Alignment-Aware Acoustic-Text Pretraining (A3T), which reconstructs masked acoustic signals with text input and acoustic-text alignment during training. In this way, the pretrained model can generate high quality reconstructed spectrogram, which can be applied to the speech editing and unseen speaker TTS directly. Experiments show A3T outperforms SOTA models on speech editing, and improves multi-speaker speech synthesis without the external speaker verification model.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 24 of its 39 references that are in the map. First posted on arXiv 18 Mar 2022.

Builds on

24 of 39 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  6. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  7. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  8. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  9. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  10. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  11. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 340
  12. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 309
12 more
  1. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 282
  2. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 259
  3. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 215
  4. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 188
  5. 2020Recent Developments on ESPnet Toolkit Boosted by Conformercited by 132
  6. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
  7. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  8. 2021Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translationcited by 47
  9. 2021EditSpeech: A Text Based Speech Editing System Using Partial Inference and Bidirectional Fusioncited by 32
  10. 2020MAM: Masked Acoustic Modeling for End-to-End Speech-to-Text Translationcited by 13
  11. 2021Context-Aware Prosody Correction for Text-Based Speech Editingcited by 13
  12. 2019Non-Autoregressive Neural Text-to-Speechcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 11; 2024: 14; 2025: 7; 2026: 3.

40 papers.

Sort:

By area:

  1. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  2. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  3. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 361
  4. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 120
  5. 2024Proactive Detection of Voice Cloning with Localized Watermarkingcited by 78
  6. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  7. 2023SpeechX: Neural Codec Language Model as a Versatile Speech Transformercited by 64
  8. 2023Generative Pre-training for Speech with Flow Matchingcited by 55
  9. 2023Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Biascited by 51
  10. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  11. 2023FluentSpeech: Stutter-Oriented Automatic Speech Editing with Context-Aware Diffusion Modelscited by 23
  12. 2022Virtuoso: Massive Multilingual Speech-Text Joint Semi-Supervised Learning for Text-To-Speechcited by 15
28 more
  1. 2023DiffVoice: Text-to-Speech with Latent Diffusioncited by 13
  2. 2024SongCreator: Lyrics-based Universal Song Generationcited by 12
  3. 2024Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Likecited by 11
  4. 2024dMel: Speech Tokenization made Simplecited by 11
  5. 2024MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generationcited by 9
  6. 2025SpeakStream: Streaming Text-to-Speech with Interleaved Datacited by 8
  7. 2024FedTabDiff: Federated Learning of Diffusion Probabilistic Models for Synthetic Mixed-Type Tabular Data Generationcited by 7
  8. 2023FluentEditor: Text-based Speech Editing by Considering Acoustic and Prosody Consistencycited by 6
  9. 2025Voice Cloning: Comprehensive Surveycited by 4
  10. 2022Emotion Selectable End-to-End Text-based Speech Editingcited by 3
  11. 2024DiffEditor: Enhancing Speech Editing with Semantic Enrichment and Acoustic Consistencycited by 3
  12. 2025SeniorTalk: A Chinese Conversation Dataset with Rich Annotations for Super-Aged Seniorscited by 3
  13. 2023Improving Code-Switching and Named Entity Recognition in ASR with Speech Editing based Data Augmentationcited by 2
  14. 2024Mechanisms of Multimodal Synchronization: Insights from Decoder-Based Video-Text-to-Speech Synthesiscited by 2
  15. 2025C2AV-TSE: Context and Confidence-aware Audio Visual Target Speaker Extractioncited by 2
  16. 2022ALCAP: Alignment-Augmented Music Captionercited by 1
  17. 2023A Comparative Analysis Of Latent Regressor Losses For Singing Voice Conversioncited by 1
  18. 2023Cross-Utterance Conditioned VAE for Speech Generationcited by 1
  19. 2024AttentionStitch: How Attention Solves the Speech Editing Problemcited by 1
  20. 2024Speech Editing -- a Summarycited by 1
  21. 2024FluentEditor2: Text-based Speech Editing by Modeling Multi-Scale Acoustic and Prosody Consistencycited by 1
  22. 2025UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Modelscited by 1
  23. 2026Unifying Speech Editing Detection and Content Localization via Prior-Enhanced Audio LLMscited by 1
  24. 2026Prosodic Boundary-Aware Streaming Generation for LLM-Based TTS with Streaming Text Inputcited by 1
  25. 2022ERNIE-SAT: Speech and Text Joint Pretraining for Cross-Lingual Multi-Speaker Text-to-Speechcited by 0
  26. 2025Towards Emotionally Consistent Text-Based Speech Editing: Introducing EmoCorrector and The ECD-TSE Datasetcited by 0
  27. 2025Speak, Edit, Repeat: High-Fidelity Voice Editing and Zero-Shot TTS with Cross-Attentive Mambacited by 0
  28. 2026dots.tts.edit: Precisely Controlled Speech Editing with a Continuous Autoregressive Modelcited by 0