# Towards Inclusive NLP: Assessing Compressed Multilingual Transformers across Diverse Language Benchmarks

arXiv: https://arxiv.org/abs/2507.19699 · HTML: https://mlsanity.com/paper/2507.19699/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 25 Jul 2025.

## Builds on (most cited first, 23 of 23)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Structured Pruning of Large Language Models](https://mlsanity.com/paper/1910.04732/) (2019)
- [Large language models in healthcare and medical domain: A review](https://mlsanity.com/paper/2401.06775/) (2023)
- [ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic](https://mlsanity.com/paper/2402.12840/) (2024)
- [AceGPT, Localizing Large Language Models in Arabic](https://mlsanity.com/paper/2309.12053/) (2023)
- [Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models](https://mlsanity.com/paper/2308.16149/) (2023)
- [LLM Agents for Education: Advances and Applications](https://mlsanity.com/paper/2503.11733/) (2025)
- [IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages](https://mlsanity.com/paper/2404.16816/) (2024)
- [AraGPT2: Pre-Trained Transformer for Arabic Language Generation](https://mlsanity.com/paper/2012.15520/) (2020)
- [ALLaM: Large Language Models for Arabic and English](https://mlsanity.com/paper/2407.15390/) (2024)
- [Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectives](https://mlsanity.com/paper/2407.14962/) (2024)
- [Spanish and LLM Benchmarks: is MMLU Lost in Translation?](https://mlsanity.com/paper/2406.17789/) (2024)
- [MMLU-Pro+: Evaluating Higher-Order Reasoning and Shortcut Learning in LLMs](https://mlsanity.com/paper/2409.02257/) (2024)
- [Better to Ask in English: Evaluation of Large Language Models on English, Low-resource and Cross-Lingual Settings](https://mlsanity.com/paper/2410.13153/) (2024)
- [ArabLegalEval: A Multitask Benchmark for Assessing Arabic Legal Knowledge in Large Language Models](https://mlsanity.com/paper/2408.07983/) (2024)
- [Analyzing Multilingual Competency of LLMs in Multi-Turn Instruction Following: A Case Study of Arabic](https://mlsanity.com/paper/2310.14819/) (2023)
- [Do Large Language Models Speak All Languages Equally? A Comparative Study in Low-Resource Settings](https://mlsanity.com/paper/2408.02237/) (2024)
- [A Reality check of the benefits of LLM in business](https://mlsanity.com/paper/2406.10249/) (2024)
- [THaMES: An End-to-End Tool for Hallucination Mitigation and Evaluation in Large Language Models](https://mlsanity.com/paper/2409.11353/) (2024)
- [Data Augmentation using Transformers and Similarity Measures for Improving Arabic Text Classification](https://mlsanity.com/paper/2212.13939/) (2022)
- [Suvach -- Generated Hindi QA benchmark](https://mlsanity.com/paper/2404.19254/) (2024)

## Built on by (most cited first, 1 of 1)

- [When Simpler Is Better: Evaluating Translation Pipelines for Medieval Latin Manuscripts](https://mlsanity.com/paper/2607.03836/) (2026)

## Abstract (arXiv)

Although LLMs have attained significant success in high-resource languages, their capacity in low-resource linguistic environments like Kannada and Arabic is not yet fully understood. This work benchmarking the performance of multilingual and monolingual Large Language Models (LLMs) across Arabic, English, and Indic languages, with particular emphasis on the effects of model compression strategies such as pruning and quantization. Findings shows significant performance differences driven by linguistic diversity and resource availability on SOTA LLMS as BLOOMZ, AceGPT, Jais, LLaMA-2, XGLM, and AraGPT2. We find that multilingual versions of the model outperform their language-specific counterparts across the board, indicating substantial cross-lingual transfer benefits. Quantization (4-bit and 8-bit) is effective in maintaining model accuracy while promoting efficiency, but aggressive pruning significantly compromises performance, especially in bigger models. Our findings pinpoint key strategies to construct scalable and fair multilingual NLP solutions and underscore the need for interventions to address hallucination and generalization errors in the low-resource setting.