mlsanityData to 14 Oct 2026

Multimodal Attention Merging for Improved Speech Recognition and Audio Event Classification

arXiv 2312.14378 ↗First posted 22 Dec 2023cs.LG cs.SD eess.AScited by 10

Not on the map. Why?

Abstract (arXiv)

Training large foundation models using self-supervised objectives on unlabeled data, followed by fine-tuning on downstream tasks, has emerged as a standard procedure. Unfortunately, the efficacy of this approach is often constrained by both limited fine-tuning compute and scarcity in labeled downstream data. We introduce Multimodal Attention Merging (MAM), an attempt that facilitates direct knowledge transfer from attention matrices of models rooted in high resource modalities, text and images, to those in resource-constrained domains, speech and audio, employing a zero-shot paradigm. MAM reduces the relative Word Error Rate (WER) of an Automatic Speech Recognition (ASR) model by up to 6.70%, and relative classification error of an Audio Event Classification (AEC) model by 10.63%. In cases where some data/compute is available, we present Learnable-MAM, a data-driven approach to merging attention matrices, resulting in a further 2.90% relative reduction in WER for ASR and 18.42% relative reduction in AEC compared to fine-tuning.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 30 of its 41 references that are in the map. First posted on arXiv 22 Dec 2023.

Builds on

30 of 41 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2015VQA: Visual Question Answeringcited by 2,506
  9. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  10. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  11. 2022Editing Models with Task Arithmeticcited by 1,120
  12. 2021Merging Models with Fisher-Weighted Averagingcited by 621
18 more
  1. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  2. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
  3. 2019Model Fusion via Optimal Transportcited by 219
  4. 2019Generalized Sliced Wasserstein Distancescited by 208
  5. 2019Class-incremental Learning via Deep Model Consolidationcited by 173
  6. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  7. 2022Fusing finetuned models for better pretrainingcited by 110
  8. 2019Are All Layers Created Equal?cited by 94
  9. 2021Voice2Series: Reprogramming Acoustic Models for Time Series Classificationcited by 91
  10. 2023AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Modelscited by 84
  11. 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 77
  12. 2023An Empirical Study of Multimodal Model Mergingcited by 52
  13. 2023Multimodal Neurons in Pretrained Text-Only Transformerscited by 49
  14. 2022Pretraining with Artificial Language: Studying Transferable Knowledge in Language Modelscited by 28
  15. 2021On the Transferability of Pre-trained Language Models: A Study from Artificial Datasetscited by 25
  16. 2022lo-fi: distributed fine-tuning without communicationcited by 19
  17. 2023How to Estimate Model Transferability of Pre-Trained Speech Models?cited by 10
  18. 2018A Unified Framework for Domain Adaptation using Metric Learning on Manifoldscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 4; 2026: 1.

10 papers.

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  2. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  3. 2024Task Arithmetic can Mitigate Synthetic-to-Real Gap in Automatic Speech Recognitioncited by 14
  4. 2024Model Composition for Multimodal Large Language Modelscited by 12
  5. 2025Disrupting Model Merging: A Parameter-Level Defense Without Sacrificing Accuracycited by 11
  6. 2025Selective Attention Merging for low resource tasks: A case study of Child ASRcited by 6
  7. 2025To See a World in a Spark of Neuron: Disentangling Multi-task Interference for Training-free Model Mergingcited by 6
  8. 2025Multi-Modality Expansion and Retention for LLMs through Parameter Merging and Decouplingcited by 5
  9. 2024Multimodal Lego: Model Merging and Fine-Tuning Across Topologies and Modalities in Biomedicinecited by 2
  10. 2026Exploring the potential and limitations of Model Merging for Multi-Domain Adaptation in ASRcited by 1