# Jointly Improving Dialect Identification and ASR in Indian Languages using Multimodal Feature Fusion

arXiv: https://arxiv.org/abs/2607.02862 · HTML: https://mlsanity.com/paper/2607.02862/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 28 references that are in the map.
- First posted on arXiv 3 Jul 2026.

## Builds on (most cited first, 12 of 12)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Bottleneck Transformers for Visual Recognition](https://mlsanity.com/paper/2101.11605/) (2021)
- [Multilingual and code-switching ASR challenges for low resource Indian languages](https://mlsanity.com/paper/2104.00235/) (2021)
- [Towards Building ASR Systems for the Next Billion Users](https://mlsanity.com/paper/2111.03945/) (2021)
- [A Configurable Multilingual Model is All You Need to Recognize All Languages](https://mlsanity.com/paper/2107.05876/) (2021)
- [Streaming End-to-End Multilingual Speech Recognition with Joint Language Identification](https://mlsanity.com/paper/2209.06058/) (2022)
- [On the Robustness of Arabic Speech Dialect Identification](https://mlsanity.com/paper/2306.03789/) (2023)
- [Automatic Arabic Dialect Identification Systems for Written Texts: A Survey](https://mlsanity.com/paper/2009.12622/) (2020)
- [A Parameter-Efficient Learning Approach to Arabic Dialect Identification with Pre-Trained General-Purpose Speech Model](https://mlsanity.com/paper/2305.11244/) (2023)
- [Low-resource speech recognition and dialect identification of Irish in a multi-task framework](https://mlsanity.com/paper/2405.01293/) (2024)
- [North S'ami Dialect Identification with Self-supervised Speech Models](https://mlsanity.com/paper/2305.11864/) (2023)
- [DuDe: Dual-Decoder Multilingual ASR for Indian Languages using Common Label Set](https://mlsanity.com/paper/2210.16739/) (2022)

## Abstract (arXiv)

Automatic Speech Recognition (ASR) and Dialect Identification (DID) are crucial for Indian languages, many of which are low-resource and exhibit significant dialectal differences. Existing methods often optimize ASR or DID individually, resulting in performance trade-offs. In this work, we propose a multimodal framework that jointly improves ASR and DID. Our method employs a Bottleneck Encoder to extract dialectal features from Conformer-based speech representations and a RoBERTa encoder to process ASR-generated CTC embeddings. A gating mechanism merges these features, followed by an attention encoder to refine the representations. The learned embeddings are concatenated with Conformer outputs to enhance ASR features. Evaluated on eight Indian languages with thirty-three dialects, our method achieves an average DID accuracy of 81.63% and average CER and WER of 4.65% and 17.73%, respectively. These results highlight the effectiveness of our method for joint ASR-DID modeling.