mlsanityData to 14 Oct 2026

Evaluating Large Language Models along Dimensions of Language Variation: A Systematik Invesdigatiom uv Cross-lingual Generalization

arXiv 2406.13718 ↗First posted 19 Jun 2024cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

While large language models exhibit certain cross-lingual generalization capabilities, they suffer from performance degradation (PD) on unseen closely-related languages (CRLs) and dialects relative to their high-resource language neighbour (HRLN). However, we currently lack a fundamental understanding of what kinds of linguistic distances contribute to PD, and to what extent. Furthermore, studies of cross-lingual generalization are confounded by unknown quantities of CRL language traces in the training data, and by the frequent lack of availability of evaluation data in lower-resource related languages and dialects. To address these issues, we model phonological, morphological, and lexical distance as Bayesian noise processes to synthesize artificial languages that are controllably distant from the HRLN. We analyse PD as a function of underlying noise parameters, offering insights on model robustness to isolated and composed linguistic phenomena, and the impact of task and HRL characteristics on PD. We calculate parameter posteriors on real CRL-HRLN pair data and show that they follow computed trends of artificial languages, demonstrating the viability of our noisers. Our framework offers a cheap solution for estimating task performance on an unseen CRL given HRLN performance using its posteriors, as well as for diagnosing observed PD on a CRL in terms of its linguistic distances from its HRLN, and opens doors to principled methods of mitigating performance degradation.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 30 of its 42 references that are in the map. First posted on arXiv 19 Jun 2024.

Builds on

30 of 42 references.

References · 30

  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  2. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  3. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  4. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  5. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  6. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  7. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  8. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
  9. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 348
  10. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  11. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 306
  12. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
18 more
  1. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 289
  2. 2022Challenges and Strategies in Cross-Cultural NLPcited by 218
  3. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 211
  4. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 206
  5. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 190
  6. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 176
  7. 2021Evaluating the Robustness of Neural Language Models to Input Perturbationscited by 87
  8. 2023A Benchmark for Learning to Translate a New Language from One Grammar Bookcited by 75
  9. 2021It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoningcited by 69
  10. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  11. 2024LLMs Are Few-Shot In-Context Low-Resource Language Learnerscited by 68
  12. 2023Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Promptscited by 45
  13. 2023Quantifying the Dialect Gap and its Correlates Across Languagescited by 33
  14. 2024Teaching Large Language Models an Unseen Language on the Flycited by 30
  15. 2017How Robust Are Character-Based Word Embeddings in Tagging and MT Against Wrod Scramlbing or Randdm Nouse?cited by 23
  16. 2022Multi Task Learning For Zero Shot Performance Prediction of Multilingual Modelscited by 20
  17. 2024Hire a Linguist!: Learning Endangered Languages with In-Context Linguistic Descriptionscited by 12
  18. 2024Constructing and Expanding Low-Resource and Underrepresented Parallel Datasets for Indonesian Local Languagescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 3.

7 papers.

  1. 2025How Important is `Perfect' English for Machine Translation Prompts?cited by 2
  2. 2024Dialetto, ma Quanto Dialetto? Transcribing and Evaluating Dialects on a Continuumcited by 1
  3. 2025DialUp! Modeling the Language Continuum by Adapting Models to Dialects and Dialects to Modelscited by 1
  4. 2026Translating the Translator: Decomposing the Cost of English-Forced Inter-Agent Communicationcited by 1
  5. 2025Benchmarking Sociolinguistic Diversity in Swahili NLP: A Taxonomy-Guided Approachcited by 0
  6. 2026Harnessing Linguistic Dissimilarity for Language Generalization on Unseen Low-Resource Varietiescited by 0
  7. 2026Different Perturbations, Different Mechanisms: Understanding Continued Pre-training for Zero-Shot Dialect Robustnesscited by 0