# SHAMS: An Audio-Grounded Pronunciation Benchmark for Levantine Arabic

arXiv: https://arxiv.org/abs/2610.01427 · HTML: https://mlsanity.com/paper/2610.01427/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 57 references that are in the map.
- First posted on arXiv 1 Oct 2026.

## Builds on (most cited first, 24 of 24)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [SeamlessM4T: Massively Multilingual & Multimodal Machine Translation](https://mlsanity.com/paper/2308.11596/) (2023)
- [Universal Phone Recognition with a Multilingual Allophone System](https://mlsanity.com/paper/2002.11800/) (2020)
- [AraT5: Text-to-Text Transformers for Arabic Language Generation](https://mlsanity.com/paper/2109.12068/) (2021)
- [Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discrimination](https://mlsanity.com/paper/2406.08818/) (2024)
- [ByT5 model for massively multilingual grapheme-to-phoneme conversion](https://mlsanity.com/paper/2204.03067/) (2022)
- [ZIPA: A family of efficient models for multilingual phone recognition](https://mlsanity.com/paper/2505.23170/) (2025)
- [Universal Automatic Phonetic Transcription into the International Phonetic Alphabet](https://mlsanity.com/paper/2308.03917/) (2023)
- [POWSM: A Phonetic Open Whisper-Style Speech Foundation Model](https://mlsanity.com/paper/2510.24992/) (2025)
- [Neural Arabic Text Diacritization: State of the Art Results and a Novel Approach for Machine Translation](https://mlsanity.com/paper/1911.03531/) (2019)
- [Nabra: Syrian Arabic Dialects with Morphological Annotations](https://mlsanity.com/paper/2310.17315/) (2023)
- [Data Quality Issues in Multilingual Speech Datasets: The Need for Sociolinguistic Awareness and Proactive Language Planning](https://mlsanity.com/paper/2506.17525/) (2025)
- [Automatic Restoration of Diacritics for Speech Data Sets](https://mlsanity.com/paper/2311.10771/) (2023)
- [Arabic Diacritics in the Wild: Exploiting Opportunities for Improved Diacritization](https://mlsanity.com/paper/2406.05760/) (2024)
- [NADI 2025: The First Multidialectal Arabic Speech Processing Shared Task](https://mlsanity.com/paper/2509.02038/) (2025)
- [Dialectal Coverage And Generalization in Arabic Speech Recognition](https://mlsanity.com/paper/2411.05872/) (2024)
- [CATT: Character-based Arabic Tashkeel Transformer](https://mlsanity.com/paper/2407.03236/) (2024)
- [Munsit at NADI 2025 Shared Task 2: Pushing the Boundaries of Multidialectal Arabic ASR with Weakly Supervised Pretraining and Continual Supervised Fine-tuning](https://mlsanity.com/paper/2508.08912/) (2025)
- [Arab Voices: Mapping Standard and Dialectal Arabic Speech Technology](https://mlsanity.com/paper/2601.13319/) (2026)
- [Abjad AI at NADI 2025: CATT-Whisper: Multimodal Diacritic Restoration Using Text and Speech Representations](https://mlsanity.com/paper/2510.24247/) (2025)
- [Are LLMs Good Text Diacritizers? An Arabic and Yoruba Case Study](https://mlsanity.com/paper/2506.11602/) (2025)
- [ReNikud: Audio-Supervised Hebrew Grapheme-to-Phoneme Conversion](https://mlsanity.com/paper/2606.20179/) (2026)

## Abstract (arXiv)

Levantine Arabic (LA) is spoken by tens of millions of people, creating a pressing need for shared benchmarks to evaluate LA speech-language technologies. Evaluating such technology is particularly challenging given LA's internal diversity and its opaque and non-standardized orthography. We present SHAMS (SHami Annotated Multi-dialect Speech), a benchmark comprising 1,300 utterances drawn from open audio corpora, balanced across five LA varieties (Urban and Rural Palestinian, and Urban Jordanian, Lebanese, and Syrian). Each utterance is represented across four aligned tiers: audio, unvocalized orthography, diacritized text, and phonetic transcription. This structure supports evaluation of various downstream tasks such as diacritization, grapheme-to-phoneme conversion, automatic speech recognition, and audio-to-phoneme, grounded in audio and stratified by variety. We benchmark open and proprietary models across these tasks to demonstrate the utility of this benchmark for measuring progress across LA. We release SHAMS at https://shams-nlp.github.io .