# MELAC: Massive Evaluation of Large Language Models with Alignment of Culture in Persian Language

arXiv: https://arxiv.org/abs/2508.00673 · HTML: https://mlsanity.com/paper/2508.00673/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 54 references that are in the map.
- First posted on arXiv 1 Aug 2025.

## Builds on (most cited first, 30 of 30)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024)
- [Don't Make Your LLM an Evaluation Benchmark Cheater](https://mlsanity.com/paper/2311.01964/) (2023)
- [Aya 23: Open Weight Releases to Further Multilingual Progress](https://mlsanity.com/paper/2405.15032/) (2024)
- [Multilingual Constituency Parsing with Self-Attention and Pre-Training](https://mlsanity.com/paper/1812.11760/) (2018)
- [Survey of Cultural Awareness in Language Models: Text and Beyond](https://mlsanity.com/paper/2411.00860/) (2024)
- [CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean](https://mlsanity.com/paper/2403.06412/) (2024)
- [(A)I Am Not a Lawyer, But...: Engaging Legal Experts towards Responsible LLM Policies for Legal Advice](https://mlsanity.com/paper/2402.01864/) (2024)
- [(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts](https://mlsanity.com/paper/2405.11804/) (2024)
- [SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages](https://mlsanity.com/paper/2406.10118/) (2024)
- [Culture is Not Trivia: Sociocultural Theory for Cultural NLP](https://mlsanity.com/paper/2502.12057/) (2025)
- [How to Design Translation Prompts for ChatGPT: An Empirical Study](https://mlsanity.com/paper/2304.02182/) (2023)
- [CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning](https://mlsanity.com/paper/2401.14011/) (2024)
- [Towards Multilingual LLM Evaluation for European Languages](https://mlsanity.com/paper/2410.08928/) (2024)
- [Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language?](https://mlsanity.com/paper/2404.06644/) (2024)
- [JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluation](https://mlsanity.com/paper/2410.17250/) (2024)
- [What Drives Performance in Multilingual Language Models?](https://mlsanity.com/paper/2404.19159/) (2024)
- [See What LLMs Cannot Answer: A Self-Challenge Framework for Uncovering LLM Weaknesses](https://mlsanity.com/paper/2408.08978/) (2024)
- [SeaExam and SeaBench: Benchmarking LLMs with Local Multilingual Questions in Southeast Asia](https://mlsanity.com/paper/2502.06298/) (2025)
- [CamelEval: Advancing Culturally Aligned Arabic Language Models and Benchmarks](https://mlsanity.com/paper/2409.12623/) (2024)
- [IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding](https://mlsanity.com/paper/2501.15747/) (2025)
- [A Survey on sentiment analysis in Persian: A Comprehensive System Perspective Covering Challenges and Advances in Resources, and Methods](https://mlsanity.com/paper/2104.14751/) (2021)
- [naab: A ready-to-use plug-and-play corpus for Farsi](https://mlsanity.com/paper/2208.13486/) (2022)
- [Matina: A Large-Scale 73B Token Persian Text Corpus](https://mlsanity.com/paper/2502.09188/) (2025)
- [RusCode: Russian Cultural Code Benchmark for Text-to-Image Generation](https://mlsanity.com/paper/2502.07455/) (2025)

## Built on by (most cited first, 3 of 3)

- [PerHalluEval: Persian Hallucination Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2509.21104/) (2025)
- [FarsiMCQGen: a Persian Multiple-choice Question Generation Framework](https://mlsanity.com/paper/2510.15134/) (2025)
- [PersLitEval: Fine-grained Benchmark and Evaluation of LLMs on Persian Literature Questions](https://mlsanity.com/paper/2605.27015/) (2026)

## Abstract (arXiv)

As large language models (LLMs) become increasingly embedded in our daily lives, evaluating their quality and reliability across diverse contexts has become essential. While comprehensive benchmarks exist for assessing LLM performance in English, there remains a significant gap in evaluation resources for other languages. Moreover, because most LLMs are trained primarily on data rooted in European and American cultures, they often lack familiarity with non-Western cultural contexts. To address this limitation, our study focuses on the Persian language and Iranian culture. We introduce 19 new evaluation datasets specifically designed to assess LLMs on topics such as Iranian law, Persian grammar, Persian idioms, and university entrance exams. Using these datasets, we benchmarked 41 prominent LLMs, aiming to bridge the existing cultural and linguistic evaluation gap in the field.