Data to 14 Oct 2026

Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models

arXiv 2010.05874 ↗First posted 12 Oct 2020cs.CL cs.LGcited by 169

Not on the map. Why?

Abstract (arXiv)

Massively multilingual models subsuming tens or even hundreds of languages pose great challenges to multi-task optimization. While it is a common practice to apply a language-agnostic procedure optimizing a joint multilingual task objective, how to properly characterize and take advantage of its underlying problem structure for improving optimization efficiency remains under-explored. In this paper, we attempt to peek into the black-box of multilingual optimization through the lens of loss function geometry. We find that gradient similarity measured along the optimization trajectory is an important signal, which correlates well with not only language proximity but also the overall model performance. Such observation helps us to identify a critical limitation of existing gradient-based multi-task learning methods, and thus we derive a simple and scalable optimization procedure, named Gradient Vaccine, which encourages more geometrically aligned parameter updates for close tasks. Empirically, our method obtains significant model performance gains on multilingual machine translation and XTREME benchmark tasks for multilingual language models. Our work reveals the importance of properly measuring and utilizing language proximity in multilingual optimization, and has broader implications for multi-task learning beyond multilingual modeling.

Key facts. Cited by 169 papers in the map (data through 14 Oct 2026). First posted on arXiv 12 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 20; 2022: 23; 2023: 38; 2024: 37; 2025: 27; 2026: 24.

169 papers.

Sort:

By area:

2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
2023Efficient Diffusion Training via Min-SNR Weighting Strategycited by 246
2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 222
2022Multi-Task Learning as a Bargaining Gamecited by 215
2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 151
2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
2021Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learningcited by 101
2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 98
2024Representation Surgery for Multi-Task Model Mergingcited by 96
2021RotoGrad: Gradient Homogenization in Multitask Learningcited by 83
2023Modular Deep Learningcited by 83

38 more

2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
2023Independent Component Alignment for Multi-Task Learningcited by 70
2022In Defense of the Unitary Scalarization for Deep Multi-Task Learningcited by 67
2021Multi-Task Learning in Natural Language Processing: An Overviewcited by 62
2021Scaling End-to-End Models for Large-Scale Multilingual ASRcited by 61
2021Domain Generalization via Gradient Surgerycited by 60
2022Do Current Multi-Task Optimization Methods in Deep Learning Even Help?cited by 60
2024Smooth Tchebycheff Scalarization for Multi-Objective Optimizationcited by 54
2022PaCo: Parameter-Compositional Multi-Task Reinforcement Learningcited by 46
2023TaskExpert: Dynamically Assembling Multi-Task Representations with Memorial Mixture-of-Expertscited by 45
2024X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scalecited by 36
2024Fair Resource Allocation in Multi-Task Learningcited by 35
2024Make Continual Learning Stronger via C-Flatcited by 35
2022Multi-Head Adapter Routing for Cross-Task Generalizationcited by 33
2023Addressing Negative Transfer in Diffusion Modelscited by 33
2024Customizing Language Models with Instance-wise LoRA for Sequential Recommendationcited by 32
2021Improving Multilingual Translation by Representation and Gradient Regularizationcited by 31
2024Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning ratecited by 31
2022Combining Modular Skills in Multitask Learningcited by 28
2022Improving Multi-Task Generalization via Regularizing Spurious Correlationcited by 28
2023ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learningcited by 28
2023Scaling Laws for Multilingual Neural Machine Translationcited by 27
2023Dual-Balancing for Multi-Task Learningcited by 27
2022LibMTL: A Python Library for Multi-Task Learningcited by 25
2022Causes and Cures for Interference in Multilingual Translationcited by 23
2023Visual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspectivecited by 23
2023HarmonyDream: Task Harmonization Inside World Modelscited by 22
2023Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large Modelscited by 20
2021Uncertainty-Aware Balancing for Multilingual and Multi-Domain Neural Machine Translation Trainingcited by 19
2023Scalarization for Multi-Task and Multi-Domain Learning at Scalecited by 19
2024Neuron Specialization: Leveraging intrinsic task modularity for multilingual machine translationcited by 19
2021Distributionally Robust Multilingual Machine Translationcited by 18
2024DiffusionMTL: Learning Multi-Task Denoising Diffusion Model from Partially Annotated Datacited by 18
2024Task-Adaptive Pretrained Language Models via Clustered-Importance Samplingcited by 18
2025Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyondcited by 18
2023FedHCA2: Towards Hetero-Client Federated Multi-Task Learningcited by 17
2021Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learningcited by 16
2021Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Banditscited by 16

50 most cited of 169 All 169 (JSON)