# Benchmarking Open-Source Large Language Models for Persian in Zero-Shot and Few-Shot Learning

arXiv: https://arxiv.org/abs/2510.12807 · HTML: https://mlsanity.com/paper/2510.12807/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 30 references that are in the map.
- First posted on arXiv 5 Oct 2025.

## Builds on (most cited first, 13 of 13)

- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Few-shot Learning with Multilingual Language Models](https://mlsanity.com/paper/2112.10668/) (2021)
- [mGPT: Few-Shot Learners Go Multilingual](https://mlsanity.com/paper/2204.07580/) (2022)
- [Don't Trust ChatGPT when Your Question is not in English: A Study of Multilingual Abilities and Types of LLMs](https://mlsanity.com/paper/2305.16339/) (2023)
- [ParsiNLU: A Suite of Language Understanding Challenges for Persian](https://mlsanity.com/paper/2012.06154/) (2020)
- [A Survey on Large Language Models with Multilingualism: Recent Advances and New Frontiers](https://mlsanity.com/paper/2405.10936/) (2024)
- [A Survey of Useful LLM Evaluation](https://mlsanity.com/paper/2406.00936/) (2024)
- [PersianLLaMA: Towards Building First Persian Large Language Model](https://mlsanity.com/paper/2312.15713/) (2023)
- [Benchmarking Large Language Models for Persian: A Preliminary Study Focusing on ChatGPT](https://mlsanity.com/paper/2404.02403/) (2024)
- [FaMTEB: Massive Text Embedding Benchmark in Persian Language](https://mlsanity.com/paper/2502.11571/) (2025)
- [ArmanEmo: A Persian Dataset for Text-based Emotion Detection](https://mlsanity.com/paper/2207.11808/) (2022)
- [Zero-shot Sentiment Analysis in Low-Resource Languages Using a Multilingual Sentiment Lexicon](https://mlsanity.com/paper/2402.02113/) (2024)
- [We Need to Talk About Classification Evaluation Metrics in NLP](https://mlsanity.com/paper/2401.03831/) (2024)

## Built on by (most cited first, 1 of 1)

- [The Annotation Bottleneck in Persian Text NLP: Persian as an Annotation-Scarce Language](https://mlsanity.com/paper/2608.24698/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable capabilities across numerous languages; however, their effectiveness in low-resource languages like Persian requires thorough investigation. This paper presents a comprehensive benchmark of several open-source LLMs for Persian Natural Language Processing (NLP) tasks, utilizing both zero-shot and few-shot learning paradigms. We evaluate models across a range of tasks including sentiment analysis, named entity recognition, reading comprehension, and question answering, using established Persian datasets such as ParsiNLU and ArmanEmo. Our methodology encompasses rigorous experimental setups for both zero-shot and few-shot scenarios, employing metrics such as Accuracy, F1-score, BLEU, and ROUGE for performance evaluation. The results reveal that Gemma 2 consistently outperforms other models across nearly all tasks in both learning paradigms, with particularly strong performance in complex reasoning tasks. However, most models struggle with token-level understanding tasks like Named Entity Recognition, highlighting specific challenges in Persian language processing. This study contributes to the growing body of research on multilingual LLMs, providing valuable insights into their performance in Persian and offering a benchmark for future model development.