# XIFBench: Evaluating Large Language Models on Multilingual Instruction Following

arXiv: https://arxiv.org/abs/2503.07539 · HTML: https://mlsanity.com/paper/2503.07539/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 39 references that are in the map.
- First posted on arXiv 10 Mar 2025.

## Builds on (most cited first, 28 of 28)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning](https://mlsanity.com/paper/2402.06619/) (2024)
- [Benchmarking Complex Instruction-Following with Multiple Constraints Composition](https://mlsanity.com/paper/2407.03978/) (2024)
- [InFoBench: Evaluating Instruction Following Ability in Large Language Models](https://mlsanity.com/paper/2401.03601/) (2024)
- [FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models](https://mlsanity.com/paper/2310.20410/) (2023)
- [Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following](https://mlsanity.com/paper/2410.15553/) (2024)
- [Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?](https://mlsanity.com/paper/2309.07462/) (2023)
- [Multilingual Instruction Tuning With Just a Pinch of Multilinguality](https://mlsanity.com/paper/2401.01854/) (2024)
- [Can Large Language Models Understand Real-World Complex Instructions?](https://mlsanity.com/paper/2309.09150/) (2023)
- [CFBench: A Comprehensive Constraints-Following Benchmark for LLMs](https://mlsanity.com/paper/2408.01122/) (2024)
- [Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers](https://mlsanity.com/paper/2404.04925/) (2024)
- [PLUG: Leveraging Pivot Language in Cross-Lingual Instruction Tuning](https://mlsanity.com/paper/2311.08711/) (2023)
- [Benchmarking Large Language Models on Controllable Generation under Diversified Instructions](https://mlsanity.com/paper/2401.00690/) (2024)
- [X-Instruction: Aligning Language Model in Low-resource Languages with Self-curated Cross-lingual Instructions](https://mlsanity.com/paper/2405.19744/) (2024)
- [M-IFEval: Multilingual Instruction-Following Evaluation](https://mlsanity.com/paper/2502.04688/) (2025)
- [CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models](https://mlsanity.com/paper/2402.13109/) (2024)
- [Investigating Multilingual Instruction-Tuning: Do Polyglot Models Demand for Multilingual Instructions?](https://mlsanity.com/paper/2402.13703/) (2024)
- [Analyzing Multilingual Competency of LLMs in Multi-Turn Instruction Following: A Case Study of Arabic](https://mlsanity.com/paper/2310.14819/) (2023)

## Built on by (most cited first, 11 of 11)

- [Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Models](https://mlsanity.com/paper/2505.14810/) (2025)
- [Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models](https://mlsanity.com/paper/2506.01413/) (2025)
- [IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation](https://mlsanity.com/paper/2603.04738/) (2026)
- [Revisiting the Reliability of Language Models in Instruction-Following](https://mlsanity.com/paper/2512.14754/) (2025)
- [DocOS: Towards Proactive Document-Guided Actions in GUI Agents](https://mlsanity.com/paper/2605.18048/) (2026)
- [Disentangling Language Roles in Multilingual LLM Task Execution](https://mlsanity.com/paper/2605.27649/) (2026)
- [Utilizing Multilingual Encoders to Improve Large Language Models for Low-Resource Languages](https://mlsanity.com/paper/2508.09091/) (2025)
- [XLGoBench: Detecting cross-lingual skill gaps with algorithmic tasks](https://mlsanity.com/paper/2605.30788/) (2026)
- [Language Shapes Instruction Hierarchy Compliance in Multilingual LLMs](https://mlsanity.com/paper/2607.23545/) (2026)
- [Language Equality has a Price: A Systematic Investigation of Multi-turn LLM Performance for EU-24+](https://mlsanity.com/paper/2608.01395/) (2026)
- [EuroAlpaca: Task-Preserving Localisation of Instruction Data for European Languages](https://mlsanity.com/paper/2609.05043/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable instruction-following capabilities across various applications. However, their performance in multilingual settings lacks systematic investigation, with existing evaluations lacking fine-grained constraint analysis across diverse linguistic contexts. We introduce XIFBench, a comprehensive constraint-based benchmark for evaluating multilingual instruction-following abilities of LLMs, comprising 558 instructions with 0-5 additional constraints across five categories (Content, Style, Situation, Format, and Numerical) in six languages spanning different resource levels. To support reliable and consistent cross-lingual evaluation, we implement three methodological innovations: cultural accessibility annotation, constraint-level translation validation, and requirement-based evaluation using English requirements as semantic anchors across languages. Extensive experiments with various LLMs not only quantify performance disparities across resource levels but also provide detailed insights into how language resources, constraint categories, instruction complexity, and cultural specificity influence multilingual instruction-following. Our code and data are available at https://github.com/zhenyuli801/XIFBench.