mlsanityData to 13 Oct 2026

Objective Soups: Multilingual Multi-Task Modeling for Speech Processing

arXiv 2508.09228 ↗First posted 12 Aug 2025eess.AS cs.LG math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

Training a single model for multilingual, multi-task speech processing (MSP) is severely hampered by conflicting objectives between tasks like speech recognition and translation. While multi-objective optimization (MOO) aims to align gradient updates, its effectiveness diminishes as the number of tasks grows, making it difficult to find a common descent direction. This raises a fundamental question: should highly conflicting objectives be optimized jointly or separated into a hierarchical structure? To address this question, this paper investigates three multi-objective MSP formulations, which we refer to as objective soup recipes. These formulations apply multi-objective optimization at different optimization levels to mitigate potential conflicts among all objectives. To ensure efficiency, we introduce a lightweight layer-selection mechanism that computes the conflict-avoiding gradient using only the most problematic layers, minimizing computational and memory overhead. Extensive experiments on CoVoST v2, LibriSpeech, and AISHELL-1 reveal that a bi-level recipe separating recognition and translation tasks consistently outperforms standard flat optimization. Our work demonstrates that hierarchical MOO is a more effective and scalable approach for building state-of-the-art MSP models. Our code has been released at https://github.com/afmsaif/ObjectiveSoups.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 36 of its 62 references that are in the map. First posted on arXiv 12 Aug 2025.

Builds on

36 of 62 references.

References · 36

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,712
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,004
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,870
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,979
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,844
  6. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 667
  7. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 594
  8. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 446
  9. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 366
  10. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 337
  11. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 240
  12. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 159
24 more
  1. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  2. 2021Intermediate Loss Regularization for CTC-based Speech Recognitioncited by 95
  3. 2017Multilingual Speech Recognition With A Single End-To-End Modelcited by 87
  4. 2023On Penalty-based Bilevel Gradient Descent Methodcited by 87
  5. 2021Scalable and Efficient MoE Training for Multitask Multilingual Modelscited by 76
  6. 2022In Defense of the Unitary Scalarization for Deep Multi-Task Learningcited by 65
  7. 2022Do Current Multi-Task Optimization Methods in Deep Learning Even Help?cited by 60
  8. 2018Tied Multitask Learning for Neural Speech Translationcited by 54
  9. 2020Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translationcited by 54
  10. 2023Recon: Reducing Conflicting Gradients from the Root for Multi-Task Learningcited by 50
  11. 2019The stochastic multi-gradient algorithm for multi-objective optimization and its application to supervised machine learningcited by 45
  12. 2021Joint Unsupervised and Supervised Training for Multilingual ASRcited by 39
  13. 2023Three-Way Trade-Off in Multi-Objective Learning: Optimization, Generalization and Conflict-Avoidancecited by 35
  14. 2019Attention-Passing Models for Robust and Data-Efficient End-to-End Speech Translationcited by 30
  15. 2018Multilingual End-to-End Speech Recognition with A Single Transformer on Low-Resource Languagescited by 26
  16. 2020Joint Masked CPC and CTC Training for ASRcited by 20
  17. 2022Lexicographic Multi-Objective Reinforcement Learningcited by 17
  18. 2022Mitigating Gradient Bias in Multi-objective Learning: A Provably Convergent Stochastic Approachcited by 14
  19. 2020Worse WER, but Better BLEU? Leveraging Word Embedding as Intermediate in Multitask End-to-End Speech Translationcited by 12
  20. 2024FERERO: A Flexible Framework for Preference-Guided Multi-Objective Learningcited by 9
  21. 2023Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learningcited by 7
  22. 2024Joint Unsupervised and Supervised Training for Automatic Speech Recognition via Bilevel Optimizationcited by 5
  23. 2025Efficient First-Order Optimization on the Pareto Set for Multi-Objective Learning under Preference Guidancecited by 4
  24. 2024Bilevel Joint Unsupervised and Supervised Training for Automatic Speech Recognitioncited by 1

Built on by

Not cited yet.