mlsanityData to 7 Oct 2026

No Optimal Language Set Exists for Multilingual Instruction Tuning: Insights from a Linguistically-Informed Study

arXiv 2410.07809 ↗First posted 10 Oct 2024cs.CL cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Multilingual instruction tuning (MIT) is challenged by the curse of multilinguality, data scarcity, and high computational cost. A natural hypothesis is that carefully selecting a linguistically diverse set of languages yields universally better models. We test this systematically by evaluating linguistically-informed selection strategies---based on typological, geographical, semantic, and learned features---against random baselines across three model families (mGPT, mT5-xl, BLOOM) and five multilingual benchmarks. Our key negative finding is that no universal language selection strategy emerges in our fixed-budget setting: performance is strongly task- and model-dependent, and adding more languages beyond a modest threshold triggers degradation consistent with the curse of multilinguality. We discuss implications for MIT data curation and the pitfalls of benchmark-averaged evaluation. All resources are publicly available at https://github.com/GGLAB-KU/ling-informed-mit.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 14 of its 23 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

14 of 23 references.

References · 14

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  3. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  4. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  5. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  6. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  7. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  8. 2019Choosing Transfer Languages for Cross-Lingual Learningcited by 143
  9. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  10. 2020UDapter: Language Adaptation for Truly Universal Dependency Parsingcited by 77
  11. 2021It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoningcited by 69
  12. 2023Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpacacited by 60
2 more
  1. 2017Learning Language Representations for Typology Predictioncited by 40
  2. 2024Investigating Multilingual Instruction-Tuning: Do Polyglot Models Demand for Multilingual Instructions?cited by 10

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.