Incorporating Linguistic Constraints from External Knowledge Source for Audio-Visual Target Speech Extraction
Not on the map. Why?
Abstract (arXiv)
Audio-visual target speaker extraction (AV-TSE) models primarily rely on target visual cues to isolate the target speaker's voice from others. We know that humans leverage linguistic knowledge, such as syntax and semantics, to support speech perception. Inspired by this, we explore the potential of pre-trained speech-language models (PSLMs) and pre-trained language models (PLMs) as auxiliary knowledge sources for AV-TSE. In this study, we propose incorporating the linguistic constraints from PSLMs or PLMs for the AV-TSE model as additional supervision signals. Without introducing any extra computational cost during inference, the proposed approach consistently improves speech quality and intelligibility. Furthermore, we evaluate our method in multi-language settings and visual cue-impaired scenarios and show robust performance gains.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 26 of its 33 references that are in the map. First posted on arXiv 11 Jun 2025.
Builds on
26 of 33 references.
References · 26
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
- 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
- 2018SDR - half-baked or well done?cited by 423
- 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 308
- 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
- 2023Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoringcited by 42
- 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 40
- 2024SpeechBERTScore: Reference-Aware Automatic Evaluation of Speech Generation Leveraging NLP Evaluation Metricscited by 38
- 2024How Should We Extract Discrete Audio Tokens from Self-Supervised Models?cited by 24
14 more
- 2020Muse: Multi-modal target speaker extraction with visual cuescited by 21
- 2024Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructionscited by 17
- 2023AV-SepFormer: Cross-Attention SepFormer for Audio-Visual Target Speaker Extractioncited by 15
- 2025ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modelingcited by 12
- 2023Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillationcited by 11
- 2024Separate in the Speech Chain: Cross-Modal Conditional Audio-Visual Target Speech Extractioncited by 7
- 2024Transferring Knowledge from Large Foundation Models to Small Downstream Modelscited by 7
- 2024LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codeccited by 7
- 2022ImagineNET: Target Speaker Extraction with Intermittent Visual Cue through Embedding Inpaintingcited by 5
- 2024Probing Self-supervised Learning Models with Target Speech Extractioncited by 5
- 2024Target Speech Extraction with Pre-trained AV-HuBERT and Mask-And-Recover Strategycited by 5
- 2023Intuitive Multilingual Audio-Visual Speech Recognition with a Single-Trained Modelcited by 4
- 2024Target Speech Extraction with Pre-trained Self-supervised Learning Modelscited by 4
- 2025C2AV-TSE: Context and Confidence-aware Audio Visual Target Speaker Extractioncited by 2
Built on by
Citing papers per year: 2025: 1.
1 paper.