mlsanityData to 14 Oct 2026

Leveraging LLM and Self-Supervised Training Models for Speech Recognition in Chinese Dialects: A Comparative Analysis

arXiv 2505.21138 ↗First posted 27 May 2025cs.CL cs.SD eess.AScited by 5

Not on the map. Why?

Abstract (arXiv)

Large-scale training corpora have significantly improved the performance of ASR models. Unfortunately, due to the relative scarcity of data, Chinese accents and dialects remain a challenge for most ASR models. Recent advancements in self-supervised learning have shown that self-supervised pre-training, combined with large language models (LLM), can effectively enhance ASR performance in low-resource scenarios. We aim to investigate the effectiveness of this paradigm for Chinese dialects. Specifically, we pre-train a Data2vec2 model on 300,000 hours of unlabeled dialect and accented speech data and do alignment training on a supervised dataset of 40,000 hours. Then, we systematically examine the impact of various projectors and LLMs on Mandarin, dialect, and accented speech recognition performance under this paradigm. Our method achieved SOTA results on multiple dialect datasets, including Kespeech. We will open-source our work to promote reproducible research

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 30 of its 35 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

30 of 35 references.

References · 30

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  9. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  10. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  11. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  12. 2024Qwen2-Audio Technical Reportcited by 560
18 more
  1. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  2. 2023Scaling Speech Technology to 1,000+ Languagescited by 411
  3. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
  4. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 373
  5. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
  6. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 231
  7. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 197
  8. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 162
  9. 2023FunASR: A Fundamental End-to-End Speech Recognition Toolkitcited by 95
  10. 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 88
  11. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 80
  12. 2023Zipformer: A faster and better encoder for automatic speech recognitioncited by 80
  13. 2023Can Generative Large Language Models Perform ASR Error Correction?cited by 57
  14. 2022WeNet 2.0: More Productive End-to-End Speech Recognition Toolkitcited by 47
  15. 2023DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learningcited by 37
  16. 2023Leveraging Large Language Models for Exploiting ASR Uncertaintycited by 15
  17. 2024Unveiling the Potential of LLM-Based ASR on Chinese Open-Source Datasetscited by 13
  18. 2024MMGER: Multi-modal and Multi-granularity Generative Error Correction with LLM for Joint Accent and Speech Recognitioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Accent-Invariant Automatic Speech Recognition via Saliency-Driven Spectrogram Maskingcited by 2
  2. 2025PAC: Pronunciation-Aware Contextualized Large Language Model-based Automatic Speech Recognitioncited by 1
  3. 2026Towards Comprehensive Semantic Speech Embeddings for Chinese Dialectscited by 1
  4. 2026PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialectscited by 1
  5. 2026NADI 2026: The Second Multidialectal Arabic Speech Processing Shared Taskcited by 0