mlsanityData to 14 Oct 2026

Multilingual GSM-Symbolic: What determines capability transfer across languages?

arXiv 2610.03367 ↗First posted 2 Oct 2026cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We understand little about how capabilities acquired in one language carry over to another, or what governs this transfer: evaluations rely on incomparable, saturation-prone datasets and rarely examine its determinants jointly. Identifying what predicts transfer would let us avoid exhaustive evaluation across all language pairs and let developers target the factors that limit performance in low-resource languages. To evaluate cross-lingual capability transfer, we introduce Multilingual GSM-Symbolic, an extensible multilingual mathematical dataset covering 30,000 item-matched question-answer pairs and spanning 15 languages. It utilises symbolic templates to prevent overfitting and ensure generalisation by allowing generation of millions of high-quality variations from a single sample. Using Multilingual GSM-Symbolic, we quantify the largest determinants of capability as model size (β = 1.77), language resource level (β = 0.77), reasoning (β = 0.67) and typological distance (β = -0.25). This joint estimation allows these determinants to be expressed in terms of one another: a 32B model evaluated in Marathi performs like a 10B model in English. Our findings have important implications for model developers, showing that model size and reasoning narrow the performance gap between low- and high-resource languages (β = -0.27 and β = -0.20, respectively), while similar levers have little or no effect on typologically distant languages. Overall, our analysis framework explains 92% of between-language variation, but only 23% of the model-by-language variation, and predicts a model's performance on an unseen language within 6.0pp (r=.96). Incorporating measurements from just 10 templates in the target language reduces this to 4.19pp, enabling reasonable estimates of performance with little or no downstream dataset.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 40 references that are in the map. First posted on arXiv 2 Oct 2026.

Builds on

19 of 40 references.

References · 19

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023Let's Verify Step by Stepcited by 3,789
  4. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  5. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  6. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  7. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  8. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 289
  9. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 243
  10. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 206
  11. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 191
  12. 2024EuroLLM: Multilingual Language Models for Europecited by 106
7 more
  1. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 92
  2. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 77
  3. 2025Answer Matching Outperforms Multiple Choice for Language Model Evaluationcited by 41
  4. 2024Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Taskscited by 34
  5. 2025Long Chain-of-Thought Reasoning Across Languagescited by 19
  6. 2025Dynaword: From One-shot to Continuously Developed Datasetscited by 3
  7. 2026MGSM-Pro: A Simple Strategy for Robust Multilingual Mathematical Reasoning Evaluationcited by 3

Built on by

Not cited yet.