mlsanityData to 7 Oct 2026

Practice of the conformer enhanced AUDIO-VISUAL HUBERT on Mandarin and English

arXiv 2303.12187 ↗First posted 28 Feb 2023eess.AS cs.AI cs.SDcited by 1

Not on the map. Why?

Abstract (arXiv)

Considering the bimodal nature of human speech perception, lips, and teeth movement has a pivotal role in automatic speech recognition. Benefiting from the correlated and noise-invariant visual information, audio-visual recognition systems enhance robustness in multiple scenarios. In previous work, audio-visual HuBERT appears to be the finest practice incorporating modality knowledge. This paper outlines a mixed methodology, named conformer enhanced AV-HuBERT, boosting the AV-HuBERT system's performance a step further. Compared with baseline AV-HuBERT, our method in the one-phase evaluation of clean and noisy conditions achieves 7% and 16% relative WER reduction on the English AVSR benchmark dataset LRS3. Furthermore, we establish a novel 1000h Mandarin AVSR dataset CSTS. On top of the baseline AV-HuBERT, we exceed the WeNet ASR system by 14% and 18% relatively on MISP and CMLR by pre-training with this dataset. The conformer-enhanced AV-HuBERT we proposed brings 7% on MISP and 6% CER reduction on CMLR, compared with the baseline AV-HuBERT system.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 21 of its 27 references that are in the map. First posted on arXiv 28 Feb 2023.

Builds on

21 of 27 references.

References · 21

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  4. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  5. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  6. 2016Language Modeling with Gated Convolutional Networkscited by 688
  7. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  8. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 400
  9. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  10. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  11. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
  12. 2015End-to-End Attention-based Large Vocabulary Speech Recognitioncited by 148
9 more
  1. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 72
  2. 2017Combining Residual Networks with LSTMs for Lipreadingcited by 66
  3. 2020Audio-visual Recognition of Overlapped speech for the LRS2 datasetcited by 38
  4. 2020Super-Human Performance in Online Low-latency Recognition of Conversational Speechcited by 20
  5. 2018Attention-based Audio-Visual Fusion for Robust Automatic Speech Recognitioncited by 18
  6. 2021Multi-Channel Transformer Transducer for Speech Recognitioncited by 8
  7. 2020How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognitioncited by 7
  8. 2022Improving Mandarin Speech Recogntion with Block-augmented Transformercited by 6
  9. 2021Fusing information streams in end-to-end audio-visual speech recognitioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026FiLoRA: Focus-and-Ignore LoRA for Controllable Feature Reliancecited by 1