# ParsiNLU: A Suite of Language Understanding Challenges for Persian

arXiv: https://arxiv.org/abs/2012.06154 · HTML: https://mlsanity.com/paper/2012.06154/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Dec 2020.

## Built on by (most cited first, 30 of 42)

- [XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation](https://mlsanity.com/paper/2104.07412/) (2021)
- [MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages](https://mlsanity.com/paper/2204.08582/) (2022)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [KLUE: Korean Language Understanding Evaluation](https://mlsanity.com/paper/2105.09680/) (2021)
- [BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer](https://mlsanity.com/paper/2305.14857/) (2023)
- [Natural Language Reasoning, A Survey](https://mlsanity.com/paper/2303.14725/) (2023)
- [Beyond Counting Datasets: A Survey of Multilingual Dataset Construction and Necessary Resources](https://mlsanity.com/paper/2211.15649/) (2022)
- [Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language?](https://mlsanity.com/paper/2404.06644/) (2024)
- [ScandEval: A Benchmark for Scandinavian Natural Language Processing](https://mlsanity.com/paper/2304.00906/) (2023)
- [This is the way: designing and compiling LEPISZCZE, a comprehensive NLP benchmark for Polish](https://mlsanity.com/paper/2211.13112/) (2022)
- [Benchmarking Large Language Models for Persian: A Preliminary Study Focusing on ChatGPT](https://mlsanity.com/paper/2404.02403/) (2024)
- [FaBERT: Pre-training BERT on Persian Blogs](https://mlsanity.com/paper/2402.06617/) (2024)
- [FaMTEB: Massive Text Embedding Benchmark in Persian Language](https://mlsanity.com/paper/2502.11571/) (2025)
- [LMTurk: Few-Shot Learners as Crowdsourcing Workers in a Language-Model-as-a-Service Framework](https://mlsanity.com/paper/2112.07522/) (2021)
- [PerCul: A Story-Driven Cultural Evaluation of LLMs in Persian](https://mlsanity.com/paper/2502.07459/) (2025)
- [PersianMind: A Cross-Lingual Persian-English Large Language Model](https://mlsanity.com/paper/2401.06466/) (2024)
- [TookaBERT: A Step Forward for Persian NLU](https://mlsanity.com/paper/2407.16382/) (2024)
- [Multilingual Prompt Engineering in Large Language Models: A Survey Across NLP Tasks](https://mlsanity.com/paper/2505.11665/) (2025)
- [A Survey on non-English Question Answering Dataset](https://mlsanity.com/paper/2112.13634/) (2021)
- [ViMMRC 2.0 -- Enhancing Machine Reading Comprehension on Vietnamese Literature Text](https://mlsanity.com/paper/2303.18162/) (2023)
- [Investigating Transfer Learning in Multilingual Pre-trained Language Models through Chinese Natural Language Inference](https://mlsanity.com/paper/2106.03983/) (2021)
- [Leveraging Online Data to Enhance Medical Knowledge in a Small Persian Language Model](https://mlsanity.com/paper/2505.16000/) (2025)
- [Isolating Culture Neurons in Multilingual Large Language Models](https://mlsanity.com/paper/2508.02241/) (2025)
- [OPI at SemEval 2023 Task 1: Image-Text Embeddings and Multimodal Information Retrieval for Visual Word Sense Disambiguation](https://mlsanity.com/paper/2304.07127/) (2023)
- [skLEP: A Slovak General Language Understanding Benchmark](https://mlsanity.com/paper/2506.21508/) (2025)
- [ARMAN: Pre-training with Semantically Selecting and Reordering of Sentences for Persian Abstractive Summarization](https://mlsanity.com/paper/2109.04098/) (2021)
- [Empowering Persian LLMs for Instruction Following: A Novel Dataset and Training Approach](https://mlsanity.com/paper/2407.11186/) (2024)
- [PsychoLex: Unveiling the Psychological Mind of Large Language Models](https://mlsanity.com/paper/2408.08848/) (2024)
- [Advancing Retrieval-Augmented Generation for Persian: Development of Language Models, Comprehensive Benchmarks, and Best Practices for Optimization](https://mlsanity.com/paper/2501.04858/) (2025)
- [ELAB: Extensive LLM Alignment Benchmark in Persian Language](https://mlsanity.com/paper/2504.12553/) (2025)

## Abstract (arXiv)

Despite the progress made in recent years in addressing natural language understanding (NLU) challenges, the majority of this progress remains to be concentrated on resource-rich languages like English. This work focuses on Persian language, one of the widely spoken languages in the world, and yet there are few NLU datasets available for this rich language. The availability of high-quality evaluation datasets is a necessity for reliable assessment of the progress on different NLU tasks and domains. We introduce ParsiNLU, the first benchmark in Persian language that includes a range of high-level tasks -- Reading Comprehension, Textual Entailment, etc. These datasets are collected in a multitude of ways, often involving manual annotations by native speakers. This results in over 14.5k new instances across 6 distinct NLU tasks. Besides, we present the first results on state-of-the-art monolingual and multi-lingual pre-trained language-models on this benchmark and compare them with human performance, which provides valuable insights into our ability to tackle natural language understanding challenges in Persian. We hope ParsiNLU fosters further research and advances in Persian language understanding.