mlsanityData to 14 Oct 2026

Task Arithmetic can Mitigate Synthetic-to-Real Gap in Automatic Speech Recognition

arXiv 2406.02925 ↗First posted 5 Jun 2024eess.AS cs.AI cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

Synthetic data is widely used in speech recognition due to the availability of text-to-speech models, which facilitate adapting models to previously unseen text domains. However, existing methods suffer in performance when they fine-tune an automatic speech recognition (ASR) model on synthetic data as they suffer from the distributional shift commonly referred to as the synthetic-to-real gap. In this paper, we find that task vector arithmetic is effective at mitigating this gap. Our proposed method, SYN2REAL task vector, shows an average improvement of 10.03% improvement in word error rate over baselines on the SLURP dataset. Additionally, we show that an average of SYN2REAL task vectors, when we have real speeches from multiple different domains, can further adapt the original ASR model to perform better on the target text domain.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 17 of its 34 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

17 of 34 references.

References · 17

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  5. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  6. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  7. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 198
  8. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 176
  9. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 112
  10. 2023Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languagescited by 54
  11. 2023An Empirical Study of Multimodal Model Mergingcited by 52
  12. 2020Using Synthetic Audio to Improve The Recognition of Out-Of-Vocabulary Words in End-To-End ASR Systemscited by 41
5 more
  1. 2023Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentationcited by 31
  2. 2023Elastic Weight Removal for Faithful and Abstractive Dialogue Generationcited by 29
  3. 2023Text-only domain adaptation for end-to-end ASR using integrated text-to-mel-spectrogram generatorcited by 11
  4. 2023Text is All You Need: Personalizing ASR Models using Controllable Speech Synthesiscited by 10
  5. 2023Multimodal Attention Merging for Improved Speech Recognition and Audio Event Classificationcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 6.

14 papers.

Sort:

By area:

  1. 2025A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Datacited by 7
  2. 2025Selective Attention Merging for low resource tasks: A case study of Child ASRcited by 6
  3. 2025BADTV: Unveiling Backdoor Threats in Third-Party Task Vectorscited by 3
  4. 2025Task Vector in TTS: Toward Emotionally Expressive Dialectal Speech Synthesiscited by 3
  5. 2025A correlation-permutation approach for speech-music encoders model mergingcited by 2
  6. 2025SAGE: Spliced-Audio Generated Data for Enhancing Foundational Models in Low-Resource Arabic-English Code-Switched Speech Recognitioncited by 2
  7. 2026Bus-Conditioned Zero-Shot Trajectory Generation via Task Arithmeticcited by 2
  8. 2024Hard-Synth: Synthesizing Diverse Hard Samples for ASR using Zero-Shot TTS and LLMcited by 1
  9. 2025DisTaC: Conditioning Task Vectors via Distillation for Robust Model Mergingcited by 1
  10. 2026How to Leverage Synthetic Speech for LLM-Based ASR Systems?cited by 1
  11. 2026Zero-Shot Synthetic-to-Real Handwritten Text Recognition via Task Analogiescited by 0
  12. 2026When Synthetic Speech Is All You Have: Better Call GRPOcited by 0
2 more
  1. 2026Merging the Knowledge of LLMs for Automatic Speech Recognitioncited by 0
  2. 2026A Practical Recipe for Semi-Supervised Federated ASR: Online Pseudo-Labels with Server Update Stabilizationcited by 0