# Crosslingual Generalization through Multitask Finetuning

arXiv: https://arxiv.org/abs/2211.01786 · HTML: https://mlsanity.com/paper/2211.01786/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Nov 2022.

## Built on by (most cited first, 26 of 26)

- [OLMoE: Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2409.02060/) (2024)
- [Challenges and Applications of Large Language Models](https://mlsanity.com/paper/2307.10169/) (2023)
- [Generative Representational Instruction Tuning](https://mlsanity.com/paper/2402.09906/) (2024)
- [Observational Scaling Laws and the Predictability of Language Model Performance](https://mlsanity.com/paper/2405.10938/) (2024)
- [COPAL-ID: Indonesian Language Reasoning with Local Culture and Nuances](https://mlsanity.com/paper/2311.01012/) (2023)
- [Interactive Natural Language Processing](https://mlsanity.com/paper/2305.13246/) (2023)
- [Under the Surface: Tracking the Artifactuality of LLM-Generated Data](https://mlsanity.com/paper/2401.14698/) (2024)
- [Large Language Models Are Cross-Lingual Knowledge-Free Reasoners](https://mlsanity.com/paper/2406.16655/) (2024)
- [LangGPT: Rethinking Structured Reusable Prompt Design Framework for LLMs from the Programming Language](https://mlsanity.com/paper/2402.16929/) (2024)
- [Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directions](https://mlsanity.com/paper/2503.16585/) (2025)
- [Eliciting Better Multilingual Structured Reasoning from LLMs through Code](https://mlsanity.com/paper/2403.02567/) (2024)
- [Empowering Multi-step Reasoning across Languages via Tree-of-Thoughts](https://mlsanity.com/paper/2311.08097/) (2023)
- [Gained in Translation: Privileged Pairwise Judges Enhance Multilingual Reasoning](https://mlsanity.com/paper/2601.18722/) (2026)
- [Conic10K: A Challenging Math Problem Understanding and Reasoning Dataset](https://mlsanity.com/paper/2311.05113/) (2023)
- [Multilingual Test-Time Scaling via Initial Thought Transfer](https://mlsanity.com/paper/2505.15508/) (2025)
- [DiPT: Enhancing LLM reasoning through diversified perspective-taking](https://mlsanity.com/paper/2409.06241/) (2024)
- [Multi-Designated Detector Watermarking for Language Models](https://mlsanity.com/paper/2409.17518/) (2024)
- [Language Modelling Approaches to Adaptive Machine Translation](https://mlsanity.com/paper/2401.14559/) (2024)
- [K2-V2: A 360-Open, Reasoning-Enhanced LLM](https://mlsanity.com/paper/2512.06201/) (2025)
- [EVIT: Event-Oriented Instruction Tuning for Event Reasoning](https://mlsanity.com/paper/2404.11978/) (2024)
- [Towards Robust Knowledge Representations in Multilingual LLMs for Equivalence and Inheritance based Consistent Reasoning](https://mlsanity.com/paper/2410.14235/) (2024)
- [Improving Multilingual Math Reasoning for African Languages](https://mlsanity.com/paper/2505.19848/) (2025)
- [Prioritizing Informative Features and Examples for Deep Learning from Noisy Data](https://mlsanity.com/paper/2403.00013/) (2024)
- [Sentiment Reasoning for Healthcare](https://mlsanity.com/paper/2407.21054/) (2024)
- [ICPR 2024 Competition on Multilingual Claim-Span Identification](https://mlsanity.com/paper/2411.19579/) (2024)
- [Adapting English Quality Classifiers for Multilingual LLM Pretraining Data Selection](https://mlsanity.com/paper/2610.11585/) (2026)

## Abstract (arXiv)

Multitask prompted finetuning (MTF) has been shown to help large language models generalize to new tasks in a zero-shot setting, but so far explorations of MTF have focused on English data and models. We apply MTF to the pretrained multilingual BLOOM and mT5 model families to produce finetuned variants called BLOOMZ and mT0. We find finetuning large multilingual language models on English tasks with English prompts allows for task generalization to non-English languages that appear only in the pretraining corpus. Finetuning on multilingual tasks with English prompts further improves performance on English and non-English tasks leading to various state-of-the-art zero-shot results. We also investigate finetuning on multilingual tasks with prompts that have been machine-translated from English to match the language of each dataset. We find training on these machine-translated prompts leads to better performance on human-written prompts in the respective languages. Surprisingly, we find models are capable of zero-shot generalization to tasks in languages they have never intentionally seen. We conjecture that the models are learning higher-level capabilities that are both task- and language-agnostic. In addition, we introduce xP3, a composite of supervised datasets in 46 languages with English and machine-translated prompts. Our code, datasets and models are freely available at https://github.com/bigscience-workshop/xmtf.