# On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment

arXiv: https://arxiv.org/abs/2010.03017 · HTML: https://mlsanity.com/paper/2010.03017/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2020.

## Built on by (most cited first, 21 of 21)

- [Composable Sparse Fine-Tuning for Cross-Lingual Transfer](https://mlsanity.com/paper/2110.07560/) (2021)
- [Modular Deep Learning](https://mlsanity.com/paper/2302.11529/) (2023)
- [Counter-Interference Adapter for Multilingual Machine Translation](https://mlsanity.com/paper/2104.08154/) (2021)
- [Improving Multilingual Translation by Representation and Gradient Regularization](https://mlsanity.com/paper/2109.04778/) (2021)
- [Discovering Language-neutral Sub-networks in Multilingual Language Models](https://mlsanity.com/paper/2205.12672/) (2022)
- [Language Versatilists vs. Specialists: An Empirical Revisiting on Multilingual Transfer Ability](https://mlsanity.com/paper/2306.06688/) (2023)
- [Focus on the Common Good: Group Distributional Robustness Follows](https://mlsanity.com/paper/2110.02619/) (2021)
- [Distributionally Robust Multilingual Machine Translation](https://mlsanity.com/paper/2109.04020/) (2021)
- [Stepmothers are mean and academics are pretentious: What do pretrained language models learn about you?](https://mlsanity.com/paper/2109.10052/) (2021)
- [Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learning](https://mlsanity.com/paper/2110.02600/) (2021)
- [A Dataset and Baselines for Multilingual Reply Suggestion](https://mlsanity.com/paper/2106.02017/) (2021)
- [Comparison between parameter-efficient techniques and full fine-tuning: A case study on multilingual news article classification](https://mlsanity.com/paper/2308.07282/) (2023)
- [Adaptive Sparse Transformer for Multilingual Translation](https://mlsanity.com/paper/2104.07358/) (2021)
- [Low-Rank and Sparse Model Merging for Multi-Lingual Speech Recognition and Translation](https://mlsanity.com/paper/2502.17380/) (2025)
- [Pay Better Attention to Attention: Head Selection in Multilingual and Multi-Domain Sequence Modeling](https://mlsanity.com/paper/2106.10840/) (2021)
- [Family of Origin and Family of Choice: Massively Parallel Lexiconized Iterative Pretraining for Severely Low Resource Machine Translation](https://mlsanity.com/paper/2104.05848/) (2021)
- [Investigating Post-pretraining Representation Alignment for Cross-Lingual Question Answering](https://mlsanity.com/paper/2109.12028/) (2021)
- [Data-Efficient Cross-Lingual Transfer with Language-Specific Subnetworks](https://mlsanity.com/paper/2211.00106/) (2022)
- [A Conditional Generative Matching Model for Multi-lingual Reply Suggestion](https://mlsanity.com/paper/2109.07046/) (2021)
- [Do language models make human-like predictions about the coreferents of Italian anaphoric zero pronouns?](https://mlsanity.com/paper/2208.14554/) (2022)
- [Compositional Meta-Learning for Mitigating Task Heterogeneity in Physics-Informed Neural Networks](https://mlsanity.com/paper/2604.26999/) (2026)

## Abstract (arXiv)

Modern multilingual models are trained on concatenated text from multiple languages in hopes of conferring benefits to each (positive transfer), with the most pronounced benefits accruing to low-resource languages. However, recent work has shown that this approach can degrade performance on high-resource languages, a phenomenon known as negative interference. In this paper, we present the first systematic study of negative interference. We show that, contrary to previous belief, negative interference also impacts low-resource languages. While parameters are maximally shared to learn language-universal structures, we demonstrate that language-specific parameters do exist in multilingual models and they are a potential cause of negative interference. Motivated by these observations, we also present a meta-learning algorithm that obtains better cross-lingual transferability and alleviates negative interference, by adding language-specific layers as meta-parameters and training them in a manner that explicitly improves shared layers' generalization on all languages. Overall, our results show that negative interference is more common than previously known, suggesting new directions for improving multilingual representations.