# Selective HuBERT: Self-Supervised Pre-Training for Target Speaker in Clean and Mixture Speech

arXiv: https://arxiv.org/abs/2311.04526 · HTML: https://mlsanity.com/paper/2311.04526/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Nov 2023.

## Built on by (most cited first, 3 of 3)

- [Speech Separation with Pretrained Frontend to Minimize Domain Mismatch](https://mlsanity.com/paper/2411.03085/) (2024)
- [Causal Self-supervised Pretrained Frontend with Predictive Code for Speech Separation](https://mlsanity.com/paper/2504.02302/) (2025)
- [MSR-HuBERT: Self-supervised Pre-training for Adaptation to Multiple Sampling Rates](https://mlsanity.com/paper/2603.23048/) (2026)

## Abstract (arXiv)

Self-supervised pre-trained speech models were shown effective for various downstream speech processing tasks. Since they are mainly pre-trained to map input speech to pseudo-labels, the resulting representations are only effective for the type of pre-train data used, either clean or mixture speech. With the idea of selective auditory attention, we propose a novel pre-training solution called Selective-HuBERT, or SHuBERT, which learns the selective extraction of target speech representations from either clean or mixture speech. Specifically, SHuBERT is trained to predict pseudo labels of a target speaker, conditioned on an enrolled speech from the target speaker. By doing so, SHuBERT is expected to selectively attend to the target speaker in a complex acoustic environment, thus benefiting various downstream tasks. We further introduce a dual-path training strategy and use the cross-correlation constraint between the two branches to encourage the model to generate noise-invariant representation. Experiments on SUPERB benchmark and LibriMix dataset demonstrate the universality and noise-robustness of SHuBERT. Furthermore, we find that our high-quality representation can be easily integrated with conventional supervised learning methods to achieve significant performance, even under extremely low-resource labeled data.