mlsanityData to 14 Oct 2026

Disentangling Language Roles in Multilingual LLM Task Execution

arXiv 2605.27649 ↗First posted 26 May 2026cs.CL cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Multilingual LLMs are increasingly used when instruction, source content, and required response languages do not coincide. Existing benchmarks have expanded multilingual instruction-following evaluation, but they rarely isolate these three roles within a fully crossed design. We introduce MTM-Bench, a controlled benchmark for language-conditioned task execution in which each instance is defined by a triplet ((Linstr, Lcontent, Lresp)). Across English, Spanish, and Chinese, MTM-Bench enumerates all 27 triplets and contains 2,430 instances per model across semantic reversal, final-state extraction, and language purity with update realization. We evaluate 20 frontier and open-weight LLMs using decomposed metrics for semantic correctness, target-language adherence, constraint satisfaction, contamination ratio, and joint success, with scoring validated by a targeted human audit. The fully crossed design reveals that degradation is organized by the role a language occupies in the task structure, not merely by mismatch count. The response-language role is the dominant axis of variation, and a single response-slot mismatch accounts for most degradation. The response-only and full-mismatch comparison suggests that mismatch count is not a monotonic predictor of difficulty, with model-level ordering varying across systems. Task families fail through distinct channels, showing that semantic correctness alone does not capture reliable multilingual task execution.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 27 of its 44 references that are in the map. First posted on arXiv 26 May 2026.

Builds on

27 of 44 references.

References · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  9. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  10. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  11. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  12. 2019How multilingual is Multilingual BERT?cited by 645
15 more
  1. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  2. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  3. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  4. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  5. 2018Stress Test Evaluation for Natural Language Inferencecited by 213
  6. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 168
  7. 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 102
  8. 2020LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluationcited by 93
  9. 2024Understanding and Mitigating Language Confusion in LLMscited by 75
  10. 2023A Survey of Code-switching: Linguistic and Social Perspectives for Language Technologiescited by 59
  11. 2023Empowering Cross-lingual Abilities of Instruction-tuned Large Language Models by Translation-following demonstrationscited by 35
  12. 2025M-IFEval: Multilingual Instruction-Following Evaluationcited by 17
  13. 2025XIFBench: Evaluating Large Language Models on Multilingual Instruction Followingcited by 11
  14. 2025MaXIFE: Multilingual and Cross-lingual Instruction Following Evaluationcited by 7
  15. 2025CoCo-CoLa: Evaluating and Improving Language Adherence in Multilingual LLMscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.