mlsanityData to 7 Oct 2026

Selective Attention Merging for low resource tasks: A case study of Child ASR

arXiv 2501.08468 ↗First posted 14 Jan 2025cs.CL cs.SD eess.AScited by 5

Not on the map. Why?

Abstract (arXiv)

While Speech Foundation Models (SFMs) excel in various speech tasks, their performance for low-resource tasks such as child Automatic Speech Recognition (ASR) is hampered by limited pretraining data. To address this, we explore different model merging techniques to leverage knowledge from models trained on larger, more diverse speech corpora. This paper also introduces Selective Attention (SA) Merge, a novel method that selectively merges task vectors from attention matrices to enhance SFM performance on low-resource tasks. Experiments on the MyST database show significant reductions in relative word error rate of up to 14%, outperforming existing model merging and data augmentation techniques. By combining data augmentation techniques with SA Merge, we achieve a new state-of-the-art WER of 8.69 on the MyST database for the Whisper-small model, highlighting the potential of SA Merge for improving low-resource ASR.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 23 of its 39 references that are in the map. First posted on arXiv 14 Jan 2025.

Builds on

23 of 39 references.

References · 23

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  5. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  6. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2022Editing Models with Task Arithmeticcited by 1,076
  9. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  10. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  11. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  12. 2021Merging Models with Fisher-Weighted Averagingcited by 613
11 more
  1. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  2. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  3. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  4. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  5. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 148
  6. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  7. 2024Benchmarking Children's ASR with Supervised and Self-supervised Speech Foundation Modelscited by 29
  8. 2023Kid-Whisper: Towards Bridging the Performance Gap in Automatic Speech Recognition for Children VS. Adultscited by 26
  9. 2024Task Arithmetic can Mitigate Synthetic-to-Real Gap in Automatic Speech Recognitioncited by 14
  10. 2023Multimodal Attention Merging for Improved Speech Recognition and Audio Event Classificationcited by 10
  11. 2024SOA: Reducing Domain Mismatch in SSL Pipeline by Speech Only Adaptation for Low Resource ASRcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2026Mind the Shift: Using Delta SSL Embeddings to Enhance Child ASRcited by 3
  2. 2025A correlation-permutation approach for speech-music encoders model mergingcited by 2
  3. 2025K-Function: Joint Pronunciation Transcription and Feedback for Evaluating Kids Language Functioncited by 1
  4. 2026Gumbel-BEARD: Automatic Layer Selection for Self-Supervised Adaptation of Whisper in Low-Resource Domainscited by 1
  5. 2026Exploring the potential and limitations of Model Merging for Multi-Domain Adaptation in ASRcited by 0