# Deep Exploration of Cross-Lingual Zero-Shot Generalization in Instruction Tuning

arXiv: https://arxiv.org/abs/2406.08796 · HTML: https://mlsanity.com/paper/2406.08796/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 29 references that are in the map.
- First posted on arXiv 13 Jun 2024.

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples](https://mlsanity.com/paper/1903.03096/) (2019)
- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://mlsanity.com/paper/2202.01279/) (2022)
- [Few-shot Learning with Multilingual Language Models](https://mlsanity.com/paper/2112.10668/) (2021)
- [Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model](https://mlsanity.com/paper/2211.02001/) (2022)
- [Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections](https://mlsanity.com/paper/2104.04670/) (2021)
- [KLUE: Korean Language Understanding Evaluation](https://mlsanity.com/paper/2105.09680/) (2021)
- [MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer](https://mlsanity.com/paper/2109.00904/) (2021)
- [Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptation](https://mlsanity.com/paper/2305.15011/) (2023)
- [Exploring the Benefits of Training Expert Language Models over Instruction Tuning](https://mlsanity.com/paper/2302.03202/) (2023)
- [Evaluating the Zero-shot Robustness of Instruction-tuned Language Models](https://mlsanity.com/paper/2306.11270/) (2023)
- [Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation](https://mlsanity.com/paper/2205.12647/) (2022)
- [Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuning](https://mlsanity.com/paper/2305.11383/) (2023)
- [Did You Read the Instructions? Rethinking the Effectiveness of Task Definitions in Instruction Learning](https://mlsanity.com/paper/2306.01150/) (2023)
- [English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too](https://mlsanity.com/paper/2005.13013/) (2020)
- [Exploring Format Consistency for Instruction Tuning](https://mlsanity.com/paper/2307.15504/) (2023)

## Built on by (most cited first, 1 of 1)

- [A Case Study of Cross-Lingual Zero-Shot Generalization for Classical Languages in LLMs](https://mlsanity.com/paper/2505.13173/) (2025)

## Abstract (arXiv)

Instruction tuning has emerged as a powerful technique, significantly boosting zero-shot performance on unseen tasks. While recent work has explored cross-lingual generalization by applying instruction tuning to multilingual models, previous studies have primarily focused on English, with a limited exploration of non-English tasks. For an in-depth exploration of cross-lingual generalization in instruction tuning, we perform instruction tuning individually for two distinct language meta-datasets. Subsequently, we assess the performance on unseen tasks in a language different from the one used for training. To facilitate this investigation, we introduce a novel non-English meta-dataset named "KORANI" (Korean Natural Instruction), comprising 51 Korean benchmarks. Moreover, we design cross-lingual templates to mitigate discrepancies in language and instruction-format of the template between training and inference within the cross-lingual setting. Our experiments reveal consistent improvements through cross-lingual generalization in both English and Korean, outperforming baseline by average scores of 20.7% and 13.6%, respectively. Remarkably, these enhancements are comparable to those achieved by monolingual instruction tuning and even surpass them in some tasks. The result underscores the significance of relevant data acquisition across languages over linguistic congruence with unseen tasks during instruction tuning.