# Automatic Speech Recognition using Advanced Deep Learning Approaches: A survey

arXiv: https://arxiv.org/abs/2403.01255 · HTML: https://mlsanity.com/paper/2403.01255/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 145 references that are in the map.
- First posted on arXiv 2 Mar 2024.

## Builds on (most cited first, 30 of 57)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [SSAST: Self-Supervised Audio Spectrogram Transformer](https://mlsanity.com/paper/2110.09784/) (2021)
- [Transformer-based Acoustic Modeling for Hybrid Speech Recognition](https://mlsanity.com/paper/1910.09799/) (2019)
- [MAE-AST: Masked Autoencoding Audio Spectrogram Transformer](https://mlsanity.com/paper/2203.16691/) (2022)
- [Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Prompting](https://mlsanity.com/paper/2309.15649/) (2023)
- [Decentralizing Feature Extraction with Quantum Convolutional Neural Network for Automatic Speech Recognition](https://mlsanity.com/paper/2010.13309/) (2020)
- [Correction of Automatic Speech Recognition with Transformer Sequence-to-sequence Model](https://mlsanity.com/paper/1910.10697/) (2019)
- [A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks](https://mlsanity.com/paper/2010.11338/) (2020)
- [Lightweight and Efficient End-to-End Speech Recognition Using Low-Rank Transformer](https://mlsanity.com/paper/1910.13923/) (2019)
- [Training Speech Recognition Models with Federated Learning: A Quality/Cost Framework](https://mlsanity.com/paper/2010.15965/) (2020)
- [TSTNN: Two-stage Transformer based Neural Network for Speech Enhancement in the Time Domain](https://mlsanity.com/paper/2103.09963/) (2021)
- [Streaming Transformer-based Acoustic Models Using Self-attention with Augmented Memory](https://mlsanity.com/paper/2005.08042/) (2020)
- [Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modeling](https://mlsanity.com/paper/1810.03459/) (2018)
- [Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT](https://mlsanity.com/paper/2102.07594/) (2021)
- [End-to-End Speech Recognition from Federated Acoustic Models](https://mlsanity.com/paper/2104.14297/) (2021)
- [Multi-Encoder-Decoder Transformer for Code-Switching Speech Recognition](https://mlsanity.com/paper/2006.10414/) (2020)
- [Deep Transfer Learning for Automatic Speech Recognition: Towards Better Generalization](https://mlsanity.com/paper/2304.14535/) (2023)
- [CASS-NAT: CTC Alignment-based Single Step Non-autoregressive Transformer for Speech Recognition](https://mlsanity.com/paper/2010.14725/) (2020)
- [On the Usefulness of Self-Attention for Automatic Speech Recognition with Transformers](https://mlsanity.com/paper/2011.04906/) (2020)
- [Lifelong Incremental Reinforcement Learning with Online Bayesian Inference](https://mlsanity.com/paper/2007.14196/) (2020)
- [Conv-Transformer Transducer: Low Latency, Low Frame Rate, Streamable End-to-End Speech Recognition](https://mlsanity.com/paper/2008.05750/) (2020)
- [Federated Learning for Computer Vision](https://mlsanity.com/paper/2308.13558/) (2023)
- [Deep transfer learning for intrusion detection in industrial control networks: A comprehensive review](https://mlsanity.com/paper/2304.10550/) (2023)
- [Federated Acoustic Modeling For Automatic Speech Recognition](https://mlsanity.com/paper/2102.04429/) (2021)
- [Personalized Adversarial Data Augmentation for Dysarthric and Elderly Speech Recognition](https://mlsanity.com/paper/2205.06445/) (2022)
- [Deep Learning for Steganalysis of Diverse Data Types: A review of methods, taxonomy, challenges and future directions](https://mlsanity.com/paper/2308.04522/) (2023)
- [AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Model](https://mlsanity.com/paper/2308.07593/) (2023)
- [Improving Hybrid CTC/Attention End-to-end Speech Recognition with Pretrained Acoustic and Language Model](https://mlsanity.com/paper/2112.07254/) (2021)
- [Variable Attention Masking for Configurable Transformer Transducer Speech Recognition](https://mlsanity.com/paper/2211.01438/) (2022)

## Built on by (most cited first, 30 of 50)

- [Transformers and Large Language Models for Efficient Intrusion Detection Systems: A Comprehensive Survey](https://mlsanity.com/paper/2408.07583/) (2024)
- [Enhancing IoT Security with CNN and LSTM-Based Intrusion Detection Systems](https://mlsanity.com/paper/2405.18624/) (2024)
- [Advancing 3D Point Cloud Understanding through Deep Transfer Learning: A Comprehensive Survey](https://mlsanity.com/paper/2407.17877/) (2024)
- [OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Model](https://mlsanity.com/paper/2507.05177/) (2025)
- [Text-only adaptation in LLM-based ASR through text denoising](https://mlsanity.com/paper/2601.20900/) (2026)
- [Enabling ASR for Low-Resource Languages: A Comprehensive Dataset Creation Approach](https://mlsanity.com/paper/2406.01446/) (2024)
- [Whispering in Amharic: Fine-tuning Whisper for Low-resource Language](https://mlsanity.com/paper/2503.18485/) (2025)
- [Edge-ASR: Towards Low-Bit Quantization of Automatic Speech Recognition Models](https://mlsanity.com/paper/2507.07877/) (2025)
- [Artificial Intelligence for Cochlear Implants: Review of Strategies, Challenges, and Perspectives](https://mlsanity.com/paper/2403.15442/) (2024)
- [Adaptive Cyber-Attack Detection in IIoT Using Attention-Based LSTM-CNN Models](https://mlsanity.com/paper/2501.13962/) (2025)
- [Advanced Deep Learning and Large Language Models: Comprehensive Insights for Cancer Detection](https://mlsanity.com/paper/2504.13186/) (2025)
- [Munsit at NADI 2025 Shared Task 2: Pushing the Boundaries of Multidialectal Arabic ASR with Weakly Supervised Pretraining and Continual Supervised Fine-tuning](https://mlsanity.com/paper/2508.08912/) (2025)
- [StreamingClaw Technical Report](https://mlsanity.com/paper/2603.22120/) (2026)
- [Deep Transfer Learning for Kidney Cancer Diagnosis](https://mlsanity.com/paper/2408.04318/) (2024)
- [Continuous Speech Tokenizer in Text To Speech](https://mlsanity.com/paper/2410.17081/) (2024)
- [Recent Advances in Transformer and Large Language Models for UAV Applications](https://mlsanity.com/paper/2508.11834/) (2025)
- [Leveraging Transfer Learning and Mobile-enabled Convolutional Neural Networks for Improved Arabic Handwritten Character Recognition](https://mlsanity.com/paper/2509.05019/) (2025)
- [Swivuriso: The South African Next Voices Multilingual Speech Dataset](https://mlsanity.com/paper/2512.02201/) (2025)
- [Machine Learning and Transformers for Thyroid Carcinoma Diagnosis: A Review](https://mlsanity.com/paper/2403.13843/) (2024)
- [Deep Learning Techniques for Hand Vein Biometrics: A Comprehensive Review](https://mlsanity.com/paper/2409.07128/) (2024)
- [On the Robust Approximation of ASR Metrics](https://mlsanity.com/paper/2502.12408/) (2025)
- [Advancing Arabic Speech Recognition Through Large-Scale Weakly Supervised Learning](https://mlsanity.com/paper/2504.12254/) (2025)
- [Beyond Appearance: Transformer-based Person Identification from Conversational Dynamics](https://mlsanity.com/paper/2510.04753/) (2025)
- [Image and Point-cloud Classification for Jet Analysis in High-Energy Physics: A survey](https://mlsanity.com/paper/2403.11934/) (2024)
- [Recent Advances in Federated Learning Driven Large Language Models: A Survey on Architecture, Performance, and Security](https://mlsanity.com/paper/2406.09831/) (2024)
- [Sagalee: an Open Source Automatic Speech Recognition Dataset for Oromo Language](https://mlsanity.com/paper/2502.00421/) (2025)
- [Improving Pretrained YAMNet for Enhanced Speech Command Detection via Transfer Learning](https://mlsanity.com/paper/2504.19030/) (2025)
- [Enhancing Cochlear Implant Signal Coding with Scaled Dot-Product Attention](https://mlsanity.com/paper/2504.19046/) (2025)
- [PASS: Private Attributes Protection with Stochastic Data Substitution](https://mlsanity.com/paper/2506.07308/) (2025)
- [Investigating Vulnerabilities and Defenses Against Audio-Visual Attacks: A Comprehensive Survey Emphasizing Multimodal Models](https://mlsanity.com/paper/2506.11521/) (2025)

## Abstract (arXiv)

Recent advancements in deep learning (DL) have posed a significant challenge for automatic speech recognition (ASR). ASR relies on extensive training datasets, including confidential ones, and demands substantial computational and storage resources. Enabling adaptive systems improves ASR performance in dynamic environments. DL techniques assume training and testing data originate from the same domain, which is not always true. Advanced DL techniques like deep transfer learning (DTL), federated learning (FL), and reinforcement learning (RL) address these issues. DTL allows high-performance models using small yet related datasets, FL enables training on confidential data without dataset possession, and RL optimizes decision-making in dynamic environments, reducing computation costs. This survey offers a comprehensive review of DTL, FL, and RL-based ASR frameworks, aiming to provide insights into the latest developments and aid researchers and professionals in understanding the current challenges. Additionally, transformers, which are advanced DL techniques heavily used in proposed ASR frameworks, are considered in this survey for their ability to capture extensive dependencies in the input ASR sequence. The paper starts by presenting the background of DTL, FL, RL, and Transformers and then adopts a well-designed taxonomy to outline the state-of-the-art approaches. Subsequently, a critical analysis is conducted to identify the strengths and weaknesses of each framework. Additionally, a comparative study is presented to highlight the existing challenges, paving the way for future research opportunities.