mlsanityData to 7 Oct 2026

ParsiNLU: A Suite of Language Understanding Challenges for Persian

arXiv 2012.06154 ↗First posted 11 Dec 2020cs.CL cs.AIcited by 42

Not on the map. Why?

Abstract (arXiv)

Despite the progress made in recent years in addressing natural language understanding (NLU) challenges, the majority of this progress remains to be concentrated on resource-rich languages like English. This work focuses on Persian language, one of the widely spoken languages in the world, and yet there are few NLU datasets available for this rich language. The availability of high-quality evaluation datasets is a necessity for reliable assessment of the progress on different NLU tasks and domains. We introduce ParsiNLU, the first benchmark in Persian language that includes a range of high-level tasks -- Reading Comprehension, Textual Entailment, etc. These datasets are collected in a multitude of ways, often involving manual annotations by native speakers. This results in over 14.5k new instances across 6 distinct NLU tasks. Besides, we present the first results on state-of-the-art monolingual and multi-lingual pre-trained language-models on this benchmark and compare them with human performance, which provides valuable insights into our ability to tackle natural language understanding challenges in Persian. We hope ParsiNLU fosters further research and advances in Persian language understanding.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 4; 2023: 6; 2024: 7; 2025: 11; 2026: 7.

42 papers.

Sort:

By area:

  1. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 167
  2. 2022MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languagescited by 142
  3. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  4. 2021KLUE: Korean Language Understanding Evaluationcited by 113
  5. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  6. 2023Natural Language Reasoning, A Surveycited by 61
  7. 2022Beyond Counting Datasets: A Survey of Multilingual Dataset Construction and Necessary Resourcescited by 25
  8. 2024Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language?cited by 22
  9. 2023ScandEval: A Benchmark for Scandinavian Natural Language Processingcited by 16
  10. 2022This is the way: designing and compiling LEPISZCZE, a comprehensive NLP benchmark for Polishcited by 13
  11. 2024Benchmarking Large Language Models for Persian: A Preliminary Study Focusing on ChatGPTcited by 11
  12. 2024FaBERT: Pre-training BERT on Persian Blogscited by 10
30 more
  1. 2025FaMTEB: Massive Text Embedding Benchmark in Persian Languagecited by 10
  2. 2021LMTurk: Few-Shot Learners as Crowdsourcing Workers in a Language-Model-as-a-Service Frameworkcited by 8
  3. 2025PerCul: A Story-Driven Cultural Evaluation of LLMs in Persiancited by 8
  4. 2024PersianMind: A Cross-Lingual Persian-English Large Language Modelcited by 7
  5. 2024TookaBERT: A Step Forward for Persian NLUcited by 7
  6. 2021A Survey on non-English Question Answering Datasetcited by 5
  7. 2023ViMMRC 2.0 -- Enhancing Machine Reading Comprehension on Vietnamese Literature Textcited by 5
  8. 2025Multilingual Prompt Engineering in Large Language Models: A Survey Across NLP Taskscited by 5
  9. 2021Investigating Transfer Learning in Multilingual Pre-trained Language Models through Chinese Natural Language Inferencecited by 4
  10. 2025Isolating Culture Neurons in Multilingual Large Language Modelscited by 4
  11. 2023OPI at SemEval 2023 Task 1: Image-Text Embeddings and Multimodal Information Retrieval for Visual Word Sense Disambiguationcited by 3
  12. 2025Leveraging Online Data to Enhance Medical Knowledge in a Small Persian Language Modelcited by 3
  13. 2025skLEP: A Slovak General Language Understanding Benchmarkcited by 3
  14. 2021ARMAN: Pre-training with Semantically Selecting and Reordering of Sentences for Persian Abstractive Summarizationcited by 2
  15. 2024Empowering Persian LLMs for Instruction Following: A Novel Dataset and Training Approachcited by 2
  16. 2024PsychoLex: Unveiling the Psychological Mind of Large Language Modelscited by 2
  17. 2025Advancing Retrieval-Augmented Generation for Persian: Development of Language Models, Comprehensive Benchmarks, and Best Practices for Optimizationcited by 2
  18. 2025ELAB: Extensive LLM Alignment Benchmark in Persian Languagecited by 2
  19. 2023An Evaluation of Persian-English Machine Translation Datasets with Transformerscited by 1
  20. 2025Benchmarking Open-Source Large Language Models for Persian in Zero-Shot and Few-Shot Learningcited by 1
  21. 2026Unmasking the Factual-Conceptual Gap in Persian Language Modelscited by 1
  22. 2026PARSA-Bench: A Comprehensive Persian Audio-Language Model Benchmarkcited by 1
  23. 2022Persian Natural Language Inference: A Meta-learning approachcited by 0
  24. 2025PerHalluEval: Persian Hallucination Evaluation Benchmark for Large Language Modelscited by 0
  25. 2025FarsiMCQGen: a Persian Multiple-choice Question Generation Frameworkcited by 0
  26. 2026TARAZ: Persian Short-Answer Question Benchmark for Cultural Evaluation of Language Modelscited by 0
  27. 2026PersLitEval: Fine-grained Benchmark and Evaluation of LLMs on Persian Literature Questionscited by 0
  28. 2026IHUBERT: Vector-Based Semantic Deduplication and Domain-Balanced Pretraining for Persian Resourcescited by 0
  29. 2026The Annotation Bottleneck in Persian Text NLP: Persian as an Annotation-Scarce Languagecited by 0
  30. 2026TalkFa: A Unified Benchmark for Farsi Dialogue Generation and Understandingcited by 0