mlsanityData to 7 Oct 2026

Task Arithmetic for Language Expansion in Speech Translation

arXiv 2409.11274 ↗First posted 17 Sep 2024cs.CL cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Recent progress in large language models (LLMs) has gained interest in speech-text multimodal foundation models, achieving strong performance on instruction-tuned speech translation (ST). However, expanding language pairs is costly due to re-training on combined new and previous datasets. To address this, we aim to build a one-to-many ST system from existing one-to-one ST systems using task arithmetic without re-training. Direct application of task arithmetic in ST leads to language confusion; therefore, we introduce an augmented task arithmetic method incorporating a language control model to ensure correct target language generation. Our experiments on MuST-C and CoVoST-2 show BLEU score improvements of up to 4.66 and 4.92, with COMET gains of 8.87 and 11.83. In addition, we demonstrate our framework can extend to language pairs lacking paired ST training data or pre-trained ST models by synthesizing ST models based on existing machine translation (MT) and ST models via task analogies.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 28 of its 36 references that are in the map. First posted on arXiv 17 Sep 2024.

Builds on

28 of 36 references.

References · 28

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  6. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  7. 2022Editing Models with Task Arithmeticcited by 1,076
  8. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  9. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  10. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  11. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  12. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
16 more
  1. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  2. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  3. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  4. 2024Evolutionary Optimization of Model Merging Recipescited by 204
  5. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  6. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  7. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  8. 2023Joint Audio and Speech Understandingcited by 99
  9. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 77
  10. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 71
  11. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  12. 2024Diffusion Soup: Model Merging for Text-to-Image Diffusion Modelscited by 25
  13. 2024Learn and Don't Forget: Adding a New Language to ASR Foundation Modelscited by 16
  14. 2023UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructionscited by 9
  15. 2024Investigating Decoder-only Large Language Models for Speech-to-text Translationcited by 9
  16. 2024An Attribute Interpolation Method in Speech Synthesis by Model Mergingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025When Large Language Models Meet Speech: A Survey on Integration Approachescited by 9
  2. 2025Low-Rank and Sparse Model Merging for Multi-Lingual Speech Recognition and Translationcited by 6
  3. 2025Speech-FT: Merging Pre-trained And Fine-Tuned Speech Representation Models For Cross-Task Generalizationcited by 2
  4. 2025Scaling and Prompting for Improved End-to-End Spoken Grammatical Error Correctioncited by 2
  5. 2025A correlation-permutation approach for speech-music encoders model mergingcited by 2
  6. 2026Merging the Knowledge of LLMs for Automatic Speech Recognitioncited by 0