# A Parameter-Efficient Learning Approach to Arabic Dialect Identification with Pre-Trained General-Purpose Speech Model

arXiv: https://arxiv.org/abs/2305.11244 · HTML: https://mlsanity.com/paper/2305.11244/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 18 May 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [Adversarial Reprogramming of Neural Networks](https://mlsanity.com/paper/1806.11146/) (2018)
- [Voice2Series: Reprogramming Acoustic Models for Time Series Classification](https://mlsanity.com/paper/2106.09296/) (2021)
- [1.5 billion words Arabic Corpus](https://mlsanity.com/paper/1611.04033/) (2016)
- [SpeechPrompt v2: Prompt Tuning for Speech Classification Tasks](https://mlsanity.com/paper/2303.00733/) (2023)
- [Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented Speech](https://mlsanity.com/paper/2109.06952/) (2021)
- [AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Tasks](https://mlsanity.com/paper/2205.00305/) (2022)
- [Arap-Tweet: A Large Multi-Dialect Twitter Corpus for Gender, Age and Language Variety Identification](https://mlsanity.com/paper/1808.07674/) (2018)
- [From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognition](https://mlsanity.com/paper/2301.07851/) (2023)
- [Low-Resource Music Genre Classification with Cross-Modal Neural Model Reprogramming](https://mlsanity.com/paper/2211.01317/) (2022)
- [CHAPTER: Exploiting Convolutional Neural Network Adapters for Self-supervised Speech Models](https://mlsanity.com/paper/2212.01282/) (2022)
- [Automatic Dialect Detection in Arabic Broadcast Speech](https://mlsanity.com/paper/1509.06928/) (2015)
- [MIT-QCRI Arabic Dialect Identification System for the 2017 Multi-Genre Broadcast Challenge](https://mlsanity.com/paper/1709.00387/) (2017)
- [Transformer-based Arabic Dialect Identification](https://mlsanity.com/paper/2011.00699/) (2020)
- [Exploring Sequence-to-Sequence Transformer-Transducer Models for Keyword Spotting](https://mlsanity.com/paper/2211.06478/) (2022)

## Built on by (most cited first, 8 of 8)

- [Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition](https://mlsanity.com/paper/2310.06434/) (2023)
- [NADI 2024: The Fifth Nuanced Arabic Dialect Identification Shared Task](https://mlsanity.com/paper/2407.04910/) (2024)
- [It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition](https://mlsanity.com/paper/2402.05457/) (2024)
- [SpeechGen: Unlocking the Generative Power of Speech Language Models with Prompts](https://mlsanity.com/paper/2306.02207/) (2023)
- [Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Experts](https://mlsanity.com/paper/2311.01070/) (2023)
- [Efficient Compression of Multitask Multilingual Speech Models](https://mlsanity.com/paper/2405.00966/) (2024)
- [Dialect Identification Using Resource-Efficient Fine-Tuning Approaches](https://mlsanity.com/paper/2512.02074/) (2025)
- [Jointly Improving Dialect Identification and ASR in Indian Languages using Multimodal Feature Fusion](https://mlsanity.com/paper/2607.02862/) (2026)

## Abstract (arXiv)

In this work, we explore Parameter-Efficient-Learning (PEL) techniques to repurpose a General-Purpose-Speech (GSM) model for Arabic dialect identification (ADI). Specifically, we investigate different setups to incorporate trainable features into a multi-layer encoder-decoder GSM formulation under frozen pre-trained settings. Our architecture includes residual adapter and model reprogramming (input-prompting). We design a token-level label mapping to condition the GSM for Arabic Dialect Identification (ADI). This is challenging due to the high variation in vocabulary and pronunciation among the numerous regional dialects. We achieve new state-of-the-art accuracy on the ADI-17 dataset by vanilla fine-tuning. We further reduce the training budgets with the PEL method, which performs within 1.86% accuracy to fine-tuning using only 2.5% of (extra) network trainable parameters. Our study demonstrates how to identify Arabic dialects using a small dataset and limited computation with open source code and pre-trained models.