# Target Speech Extraction with Pre-trained AV-HuBERT and Mask-And-Recover Strategy

arXiv: https://arxiv.org/abs/2403.16078 · HTML: https://mlsanity.com/paper/2403.16078/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Mar 2024.

## Built on by (most cited first, 5 of 5)

- [DiffDSR: Dysarthric Speech Reconstruction Using Latent Diffusion Model](https://mlsanity.com/paper/2506.00350/) (2025)
- [C2AV-TSE: Context and Confidence-aware Audio Visual Target Speaker Extraction](https://mlsanity.com/paper/2504.00750/) (2025)
- [Incorporating Linguistic Constraints from External Knowledge Source for Audio-Visual Target Speech Extraction](https://mlsanity.com/paper/2506.09792/) (2025)
- [ELEGANCE: Efficient LLM Guidance for Audio-Visual Target Speech Extraction](https://mlsanity.com/paper/2511.06288/) (2025)
- [Real-Time Multi-Modal Embedded Vision Framework for Object Detection Facial Emotion Recognition and Biometric Identification on Low-Power Edge Platforms](https://mlsanity.com/paper/2601.11970/) (2026)

## Abstract (arXiv)

Audio-visual target speech extraction (AV-TSE) is one of the enabling technologies in robotics and many audio-visual applications. One of the challenges of AV-TSE is how to effectively utilize audio-visual synchronization information in the process. AV-HuBERT can be a useful pre-trained model for lip-reading, which has not been adopted by AV-TSE. In this paper, we would like to explore the way to integrate a pre-trained AV-HuBERT into our AV-TSE system. We have good reasons to expect an improved performance. To benefit from the inter and intra-modality correlations, we also propose a novel Mask-And-Recover (MAR) strategy for self-supervised learning. The experimental results on the VoxCeleb2 dataset show that our proposed model outperforms the baselines both in terms of subjective and objective metrics, suggesting that the pre-trained AV-HuBERT model provides more informative visual cues for target speech extraction. Furthermore, through a comparative study, we confirm that the proposed Mask-And-Recover strategy is significantly effective.