mlsanityData to 7 Oct 2026

ELEGANCE: Efficient LLM Guidance for Audio-Visual Target Speech Extraction

arXiv 2511.06288 ↗First posted 9 Nov 2025cs.SD cs.CL cs.MMcited by 0

Not on the map. Why?

Abstract (arXiv)

Audio-visual target speaker extraction (AV-TSE) models primarily rely on visual cues from the target speaker. However, humans also leverage linguistic knowledge, such as syntactic constraints, next word prediction, and prior knowledge of conversation, to extract target speech. Inspired by this observation, we propose ELEGANCE, a novel framework that incorporates linguistic knowledge from large language models (LLMs) into AV-TSE models through three distinct guidance strategies: output linguistic constraints, intermediate linguistic prediction, and input linguistic prior. Comprehensive experiments with RoBERTa, Qwen3-0.6B, and Qwen3-4B on two AV-TSE backbones demonstrate the effectiveness of our approach. Significant improvements are observed in challenging scenarios, including visual cue impaired, unseen languages, target speaker switches, increased interfering speakers, and out-of-domain test set. Demo page: https://alexwxwu.github.io/ELEGANCE/.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 42 of its 52 references that are in the map. First posted on arXiv 9 Nov 2025.

Builds on

42 of 52 references.

References · 42

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2024GPT-4o System Cardcited by 4,467
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  6. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
  7. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  8. 2018SDR - half-baked or well done?cited by 420
  9. 2018Deep Audio-Visual Speech Recognitioncited by 318
  10. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  11. 2019Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separationcited by 182
  12. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
30 more
  1. 2024An Investigation of Incorporating Mamba for Speech Enhancementcited by 61
  2. 2023Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognitioncited by 56
  3. 2023Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoringcited by 42
  4. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learnerscited by 40
  5. 2024SpeechBERTScore: Reference-Aware Automatic Evaluation of Speech Generation Leveraging NLP Evaluation Metricscited by 38
  6. 2023Neural Target Speech Extraction: An Overviewcited by 33
  7. 2024Speech Slytherin: Examining the Performance and Efficiency of Mamba for Speech Separation, Recognition, and Synthesiscited by 22
  8. 2020Muse: Multi-modal target speaker extraction with visual cuescited by 21
  9. 2024Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignmentcited by 21
  10. 2022Look&Listen: Multi-Modal Correlation Learning for Active Speaker Detection and Speech Enhancementcited by 20
  11. 2024Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructionscited by 17
  12. 2023AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Modelcited by 16
  13. 2021Selective Listening by Synchronizing Speech with Lipscited by 15
  14. 2023AV-SepFormer: Cross-Attention SepFormer for Audio-Visual Target Speaker Extractioncited by 15
  15. 2021USEV: Universal Speaker Extraction with Visual Cuecited by 11
  16. 2024Separate in the Speech Chain: Cross-Modal Conditional Audio-Visual Target Speech Extractioncited by 7
  17. 2024Listen, Chat, and Remix: Text-Guided Soundscape Remixing for Enhanced Auditory Experiencecited by 6
  18. 2022ImagineNET: Target Speaker Extraction with Intermittent Visual Cue through Embedding Inpaintingcited by 5
  19. 2024Target Speech Extraction with Pre-trained AV-HuBERT and Mask-And-Recover Strategycited by 5
  20. 2024Mamba-based Decoder-Only Approach with Bidirectional Speech Modeling for Speech Recognitioncited by 5
  21. 2025DiffDSR: Dysarthric Speech Reconstruction Using Latent Diffusion Modelcited by 5
  22. 2025EmoSLLM: Parameter-Efficient Adaptation of LLMs for Speech Emotion Recognitioncited by 4
  23. 2024Multimodal Representation Loss Between Timed Text and Audio for Regularized Speech Separationcited by 3
  24. 2025Beyond Speaker Identity: Text Guided Target Speech Extractioncited by 3
  25. 2025Bridging The Multi-Modality Gaps of Audio, Visual and Linguistic for Speech Enhancementcited by 3
  26. 2025Linguistic Knowledge Transfer Learning for Speech Enhancementcited by 3
  27. 2025Contextual Speech Extraction: Leveraging Textual History as an Implicit Cue for Target Speech Extractioncited by 2
  28. 2025C2AV-TSE: Context and Confidence-aware Audio Visual Target Speaker Extractioncited by 2
  29. 2025MeMo: Attentional Momentum for Real-Time Audio-Visual Target Speaker Extraction Under Impaired Visual Conditionscited by 2
  30. 2025Incorporating Linguistic Constraints from External Knowledge Source for Audio-Visual Target Speech Extractioncited by 1

Built on by

Not cited yet.