Disentangling Language Roles in Multilingual LLM Task Execution
Not on the map. Why?
Abstract (arXiv)
Multilingual LLMs are increasingly used when instruction, source content, and required response languages do not coincide. Existing benchmarks have expanded multilingual instruction-following evaluation, but they rarely isolate these three roles within a fully crossed design. We introduce MTM-Bench, a controlled benchmark for language-conditioned task execution in which each instance is defined by a triplet ((Linstr, Lcontent, Lresp)). Across English, Spanish, and Chinese, MTM-Bench enumerates all 27 triplets and contains 2,430 instances per model across semantic reversal, final-state extraction, and language purity with update realization. We evaluate 20 frontier and open-weight LLMs using decomposed metrics for semantic correctness, target-language adherence, constraint satisfaction, contamination ratio, and joint success, with scoring validated by a targeted human audit. The fully crossed design reveals that degradation is organized by the role a language occupies in the task structure, not merely by mismatch count. The response-language role is the dominant axis of variation, and a single response-slot mismatch accounts for most degradation. The response-only and full-mismatch comparison suggests that mismatch count is not a monotonic predictor of difficulty, with model-level ordering varying across systems. Task families fail through distinct channels, showing that semantic correctness alone does not capture reliable multilingual task execution.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 27 of its 44 references that are in the map. First posted on arXiv 26 May 2026.
Builds on
27 of 44 references.
References · 27
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
- 2019The Curious Case of Neural Text Degenerationcited by 2,916
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
- 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
- 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
- 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
- 2019How multilingual is Multilingual BERT?cited by 645
15 more
- 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
- 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
- 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
- 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
- 2018Stress Test Evaluation for Natural Language Inferencecited by 213
- 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 168
- 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 102
- 2020LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluationcited by 93
- 2024Understanding and Mitigating Language Confusion in LLMscited by 75
- 2023A Survey of Code-switching: Linguistic and Social Perspectives for Language Technologiescited by 59
- 2023Empowering Cross-lingual Abilities of Instruction-tuned Large Language Models by Translation-following demonstrationscited by 35
- 2025M-IFEval: Multilingual Instruction-Following Evaluationcited by 17
- 2025XIFBench: Evaluating Large Language Models on Multilingual Instruction Followingcited by 11
- 2025MaXIFE: Multilingual and Cross-lingual Instruction Following Evaluationcited by 7
- 2025CoCo-CoLa: Evaluating and Improving Language Adherence in Multilingual LLMscited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026What Does Post-Training Change in Multilingual Reasoning?29 Sep 2026