# Multichannel AV-wav2vec2: A Framework for Learning Multichannel Multi-Modal Speech Representation

arXiv: https://arxiv.org/abs/2401.03468 · HTML: https://mlsanity.com/paper/2401.03468/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jan 2024.

## Built on by (most cited first, 7 of 7)

- [GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translators](https://mlsanity.com/paper/2402.06894/) (2024)
- [CPT-Boosted Wav2vec2.0: Towards Noise Robust Speech Recognition for Classroom Environments](https://mlsanity.com/paper/2409.14494/) (2024)
- [Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representation](https://mlsanity.com/paper/2504.18539/) (2025)
- [Continued Pretraining for Domain Adaptation of Wav2vec2.0 in Automatic Speech Recognition for Elementary Math Classroom Settings](https://mlsanity.com/paper/2405.13018/) (2024)
- [Unify Variables in Neural Scaling Laws for General Audio Representations via Embedding Effective Rank](https://mlsanity.com/paper/2510.10948/) (2025)
- [Bootstrapping Audiovisual Speech Recognition in Zero-AV-Resource Scenarios](https://mlsanity.com/paper/2603.08249/) (2026)
- [Improving Audiovisual Speech Recognition through Synthetic Visual Data Augmentation](https://mlsanity.com/paper/2609.31961/) (2026)

## Abstract (arXiv)

Self-supervised speech pre-training methods have developed rapidly in recent years, which show to be very effective for many near-field single-channel speech tasks. However, far-field multichannel speech processing is suffering from the scarcity of labeled multichannel data and complex ambient noises. The efficacy of self-supervised learning for far-field multichannel and multi-modal speech processing has not been well explored. Considering that visual information helps to improve speech recognition performance in noisy scenes, in this work we propose a multichannel multi-modal speech self-supervised learning framework AV-wav2vec2, which utilizes video and multichannel audio data as inputs. First, we propose a multi-path structure to process multichannel audio streams and a visual stream in parallel, with intra- and inter-channel contrastive losses as training targets to fully exploit the spatiotemporal information in multichannel speech data. Second, based on contrastive learning, we use additional single-channel audio data, which is trained jointly to improve the performance of speech representation. Finally, we use a Chinese multichannel multi-modal dataset in real scenarios to validate the effectiveness of the proposed method on audio-visual speech recognition (AVSR), automatic speech recognition (ASR), visual speech recognition (VSR) and audio-visual speaker diarization (AVSD) tasks.