Hard-Synth: Synthesizing Diverse Hard Samples for ASR using Zero-Shot TTS and LLM
Not on the map. Why?
Abstract (arXiv)
Text-to-speech (TTS) models have been widely adopted to enhance automatic speech recognition (ASR) systems using text-only corpora, thereby reducing the cost of labeling real speech data. Existing research primarily utilizes additional text data and predefined speech styles supported by TTS models. In this paper, we propose Hard-Synth, a novel ASR data augmentation method that leverages large language models (LLMs) and advanced zero-shot TTS. Our approach employs LLMs to generate diverse in-domain text through rewriting, without relying on additional text data. Rather than using predefined speech styles, we introduce a hard prompt selection method with zero-shot TTS to clone speech styles that the ASR model finds challenging to recognize. Experiments demonstrate that Hard-Synth significantly enhances the Conformer model, achieving relative word error rate (WER) reductions of 6.5%/4.4% on LibriSpeech dev/test-other subsets. Additionally, we show that Hard-Synth is data-efficient and capable of reducing bias in ASR.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 30 of its 37 references that are in the map. First posted on arXiv 20 Nov 2024.
Builds on
30 of 37 references.
References · 30
- 2017Attention Is All You Needcited by 59,932
- 2023GPT-4 Technical Reportcited by 18,132
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
- 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
- 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
- 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
- 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
- 2015Attention-Based Models for Speech Recognitioncited by 400
- 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
18 more
- 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
- 2019MOSNet: Deep Learning based Objective Assessment for Voice Conversioncited by 126
- 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 118
- 2023RewriteLM: An Instruction-Tuned Large Language Model for Text Rewritingcited by 67
- 2023Can Generative Large Language Models Perform ASR Error Correction?cited by 56
- 2021Quantifying Bias in Automatic Speech Recognitioncited by 50
- 2020Using Synthetic Audio to Improve The Recognition of Out-Of-Vocabulary Words in End-To-End ASR Systemscited by 41
- 2021Synt++: Utilizing Imperfect Synthetic Data to Improve Speech Recognitioncited by 24
- 2024As Good As A Coin Toss: Human detection of AI-generated images, videos, audio, and audiovisual stimulicited by 22
- 2021Human Perception of Audio Deepfakescited by 15
- 2024Task Arithmetic can Mitigate Synthetic-to-Real Gap in Automatic Speech Recognitioncited by 14
- 2023Text-only domain adaptation for end-to-end ASR using integrated text-to-mel-spectrogram generatorcited by 11
- 2024Enhancing Low-Resource ASR through Versatile TTS: Bridging the Data Gapcited by 11
- 2022Synthesizing Dysarthric Speech Using Multi-talker TTS for Dysarthric Speech Recognitioncited by 10
- 2023Text is All You Need: Personalizing ASR Models using Controllable Speech Synthesiscited by 9
- 2024Generating Data with Text-to-Speech and Large-Language Models for Conversational Speech Recognitioncited by 9
- 2023Corpus Synthesis for Zero-shot ASR domain Adaptation using Large Language Modelscited by 5
- 2023Improving Code-Switching and Named Entity Recognition in ASR with Speech Editing based Data Augmentationcited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.