# Parameter Space Factorization for Zero-Shot Learning across Tasks and Languages

arXiv: https://arxiv.org/abs/2001.11453 · HTML: https://mlsanity.com/paper/2001.11453/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jan 2020.

## Built on by (most cited first, 22 of 22)

- [Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks](https://mlsanity.com/paper/2106.04489/) (2021)
- [Composable Sparse Fine-Tuning for Cross-Lingual Transfer](https://mlsanity.com/paper/2110.07560/) (2021)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Modular Deep Learning](https://mlsanity.com/paper/2302.11529/) (2023)
- [IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages](https://mlsanity.com/paper/2201.11732/) (2022)
- [Elastic Weight Removal for Faithful and Abstractive Dialogue Generation](https://mlsanity.com/paper/2303.17574/) (2023)
- [Combining Modular Skills in Multitask Learning](https://mlsanity.com/paper/2202.13914/) (2022)
- [HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts](https://mlsanity.com/paper/2402.12656/) (2024)
- [Hyper-X: A Unified Hypernetwork for Multi-Task Multilingual Transfer](https://mlsanity.com/paper/2205.12148/) (2022)
- [On the Calibration of Massively Multilingual Language Models](https://mlsanity.com/paper/2210.12265/) (2022)
- [Modelling Latent Translations for Cross-Lingual Transfer](https://mlsanity.com/paper/2107.11353/) (2021)
- [Interpreting Predictive Probabilities: Model Confidence or Human Label Variation?](https://mlsanity.com/paper/2402.16102/) (2024)
- [AM2iCo: Evaluating Word Meaning in Context across Low-Resource Languages with Adversarial Examples](https://mlsanity.com/paper/2104.08639/) (2021)
- [The Zeno's Paradox of `Low-Resource' Languages](https://mlsanity.com/paper/2410.20817/) (2024)
- [Minimax and Neyman-Pearson Meta-Learning for Outlier Languages](https://mlsanity.com/paper/2106.01051/) (2021)
- [Hazards from Increasingly Accessible Fine-Tuning of Downloadable Foundation Models](https://mlsanity.com/paper/2312.14751/) (2023)
- [MDAPT: Multilingual Domain Adaptive Pretraining in a Single Model](https://mlsanity.com/paper/2109.06605/) (2021)
- [Learning Label Modular Prompts for Text Classification in the Wild](https://mlsanity.com/paper/2211.17142/) (2022)
- [Transfer Learning for Multi-lingual Tasks -- a Survey](https://mlsanity.com/paper/2110.02052/) (2021)
- [Advancing Perception in Artificial Intelligence through Principles of Cognitive Science](https://mlsanity.com/paper/2310.08803/) (2023)
- [Unveiling Factors for Enhanced POS Tagging: A Study of Low-Resource Medieval Romance Languages](https://mlsanity.com/paper/2506.17715/) (2025)
- [Self Similarity Matrix based CNN Filter Pruning](https://mlsanity.com/paper/2211.01814/) (2022)

## Abstract (arXiv)

Most combinations of NLP tasks and language varieties lack in-domain examples for supervised training because of the paucity of annotated data. How can neural models make sample-efficient generalizations from task-language combinations with available data to low-resource ones? In this work, we propose a Bayesian generative model for the space of neural parameters. We assume that this space can be factorized into latent variables for each language and each task. We infer the posteriors over such latent variables based on data from seen task-language combinations through variational inference. This enables zero-shot classification on unseen combinations at prediction time. For instance, given training data for named entity recognition (NER) in Vietnamese and for part-of-speech (POS) tagging in Wolof, our model can perform accurate predictions for NER in Wolof. In particular, we experiment with a typologically diverse sample of 33 languages from 4 continents and 11 families, and show that our model yields comparable or better results than state-of-the-art, zero-shot cross-lingual transfer methods. Moreover, we demonstrate that approximate Bayesian model averaging results in smoother predictive distributions, whose entropy inversely correlates with accuracy. Hence, the proposed framework also offers robust estimates of prediction uncertainty. Our code is located at github.com/cambridgeltl/parameter-factorization