# Enhancing Quranic Learning: A Multimodal Deep Learning Approach for Arabic Phoneme Recognition

arXiv: https://arxiv.org/abs/2511.17477 · HTML: https://mlsanity.com/paper/2511.17477/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 20 references that are in the map.
- First posted on arXiv 21 Nov 2025.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data](https://mlsanity.com/paper/2101.07597/) (2021)
- [Multimodal Alignment and Fusion: A Survey](https://mlsanity.com/paper/2411.17040/) (2024)
- [An Effective End-to-End Modeling Approach for Mispronunciation Detection](https://mlsanity.com/paper/2005.08440/) (2020)
- [Towards a Unified Benchmark for Arabic Pronunciation Assessment: Quranic Recitation as Case Study](https://mlsanity.com/paper/2506.07722/) (2025)
- [An ensemble-based framework for mispronunciation detection of Arabic phonemes](https://mlsanity.com/paper/2301.01378/) (2023)
- [Mispronunciation Detection of Basic Quranic Recitation Rules using Deep Learning](https://mlsanity.com/paper/2305.06429/) (2023)
- [Arabic Multimodal Machine Learning: Datasets, Applications, Approaches, and Challenges](https://mlsanity.com/paper/2508.12227/) (2025)

## Abstract (arXiv)

Recent advances in multimodal deep learning have greatly enhanced the capability of systems for speech analysis and pronunciation assessment. Accurate pronunciation detection remains a key challenge in Arabic, particularly in the context of Quranic recitation, where subtle phonetic differences can alter meaning. Addressing this challenge, the present study proposes a transformer-based multimodal framework for Arabic phoneme mispronunciation detection that combines acoustic and textual representations to achieve higher precision and robustness. The framework integrates UniSpeech-derived acoustic embeddings with BERT-based textual embeddings extracted from Whisper transcriptions, creating a unified representation that captures both phonetic detail and linguistic context. To determine the most effective integration strategy, early, intermediate, and late fusion methods were implemented and evaluated on two datasets containing 29 Arabic phonemes, including eight hafiz sounds, articulated by 11 native speakers. Additional speech samples collected from publicly available YouTube recordings were incorporated to enhance data diversity and generalization. Model performance was assessed using standard evaluation metrics: accuracy, precision, recall, and F1-score, allowing a detailed comparison of the fusion strategies. Experimental findings show that the UniSpeech-BERT multimodal configuration provides strong results and that fusion-based transformer architectures are effective for phoneme-level mispronunciation detection. The study contributes to the development of intelligent, speaker-independent, and multimodal Computer-Aided Language Learning (CALL) systems, offering a practical step toward technology-supported Quranic pronunciation training and broader speech-based educational applications.