# PSRB: A Comprehensive Benchmark for Evaluating Persian ASR Systems

arXiv: https://arxiv.org/abs/2505.21230 · HTML: https://mlsanity.com/paper/2505.21230/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 35 references that are in the map.
- First posted on arXiv 27 May 2025.

## Builds on (most cited first, 20 of 20)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [Seamless: Multilingual Expressive and Streaming Speech Translation](https://mlsanity.com/paper/2312.05187/) (2023)
- [Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition](https://mlsanity.com/paper/2305.05084/) (2023)
- [Earnings-21: A Practical Benchmark for ASR in the Wild](https://mlsanity.com/paper/2104.11348/) (2021)
- [Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audio](https://mlsanity.com/paper/2501.11378/) (2025)
- [Hallucinations in Neural Automatic Speech Recognition: Identifying Errors and Hallucinatory Models](https://mlsanity.com/paper/2401.01572/) (2024)
- [ASR Error Correction using Large Language Models](https://mlsanity.com/paper/2409.09554/) (2024)
- [ESB: A Benchmark For Multi-Domain End-to-End Speech Recognition](https://mlsanity.com/paper/2210.13352/) (2022)
- [Adapting self-supervised models to multi-talker speech recognition using speaker embeddings](https://mlsanity.com/paper/2211.00482/) (2022)
- [Lost in Transcription, Found in Distribution Shift: Demystifying Hallucination in Speech Foundation Models](https://mlsanity.com/paper/2502.12414/) (2025)
- [Adapting Multi-Lingual ASR Models for Handling Multiple Talkers](https://mlsanity.com/paper/2305.18747/) (2023)
- [Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition System](https://mlsanity.com/paper/2407.09817/) (2024)
- [A Multi Purpose and Large Scale Speech Corpus in Persian and English for Speaker and Speech Recognition: the DeepMine Database](https://mlsanity.com/paper/1912.03627/) (2019)
- [The Balancing Act: Unmasking and Alleviating ASR Biases in Portuguese](https://mlsanity.com/paper/2402.07513/) (2024)
- [ASR in German: A Detailed Error Analysis](https://mlsanity.com/paper/2204.05617/) (2022)

## Built on by (most cited first, 2 of 2)

- [Incorporating Error Level Noise Embedding for Improving LLM-Assisted Robustness in Persian Speech Recognition](https://mlsanity.com/paper/2512.17247/) (2025)
- [Neyshekar: An Open Persian Read-Speech Corpus for Automatic Speech Recognition](https://mlsanity.com/paper/2609.14542/) (2026)

## Abstract (arXiv)

Although Automatic Speech Recognition (ASR) systems have become an integral part of modern technology, their evaluation remains challenging, particularly for low-resource languages such as Persian. This paper introduces Persian Speech Recognition Benchmark(PSRB), a comprehensive benchmark designed to address this gap by incorporating diverse linguistic and acoustic conditions. We evaluate ten ASR systems, including state-of-the-art commercial and open-source models, to examine performance variations and inherent biases. Additionally, we conduct an in-depth analysis of Persian ASR transcriptions, identifying key error types and proposing a novel metric that weights substitution errors. This metric enhances evaluation robustness by reducing the impact of minor and partial errors, thereby improving the precision of performance assessment. Our findings indicate that while ASR models generally perform well on standard Persian, they struggle with regional accents, children's speech, and specific linguistic challenges. These results highlight the necessity of fine-tuning and incorporating diverse, representative training datasets to mitigate biases and enhance overall ASR performance. PSRB provides a valuable resource for advancing ASR research in Persian and serves as a framework for developing benchmarks in other low-resource languages. A subset of the PSRB dataset is publicly available at https://huggingface.co/datasets/PartAI/PSRB.