mlsanityData to 19 Oct 2026

CF-AVSS: A Coarse-to-Fine Audio-Visual Speech Separation Framework with Target-Speaker Refinement and Post-Fusion for Real-World AVSE Challenge

arXiv 2610.21773 ↗First posted 15 Oct 2026cs.SD cs.AIcited by 0

On the map: Speech and audio language models → Speech enhancement and noise suppression

Abstract (arXiv)

Audio-visual speech enhancement (AVSE) leverages visual cues to recover target speech from mixtures corrupted by reverberation, noise, and overlapping speakers. We propose a coarse-to-fine two-stage AVSE framework. In the first stage, a progressive SPAV-TFGridNet is extended to two synchronized visual streams to produce visually ordered coarse estimates by progressively suppressing interfering speech while preserving target reverberation and shared background noise. In the second stage, a TFGridNet-based target speaker refinement network takes the original mixture and both coarse estimates as input to further recover target speech and improve signal quality. The coarse and refined estimates are finally combined using development-set-calibrated fusion. On the ISCSLP 2026 Real-World AVSE Challenge, our system tied for first place on Track 1 and ranked second on Track 2 in OVRL, while achieving the lowest CER on both tracks at 12.31% and 14.79%, respectively.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models. Builds on 20 of its 28 references that are in the map. First posted on arXiv 15 Oct 2026.

Builds on

20 of 28 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025PASE: Leveraging the Phonological Prior of WavLM for Low-Hallucination Generative Speech Enhancement17 Nov
  2. 2025Leveraging Mamba with Full-Face Vision for Audio-Visual Speech Enhancement19 Aug
  3. 2024SAV-SE: Scene-aware Audio-Visual Speech Enhancement with Selective State Space Model12 Nov
  4. 2024Robust Audio-Visual Speech Enhancement: Correcting Misassignments in Complex Environments with Advanced Post-Processing22 Sep

Other references · 16

  1. 2017Attention Is All You Needcited by 61,180
  2. 2018VoxCeleb2: Deep Speaker Recognitioncited by 1,166
  3. 2018SDR - half-baked or well done?cited by 626
  4. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 552
  5. 2020DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressorscited by 356
  6. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 319
  7. 2021DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressorscited by 292
  8. 2016Lip Reading Sentences in the Wildcited by 243
  9. 2022Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkitcited by 179
  10. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 127
  11. 2022TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separationcited by 121
  12. 2023FunASR: A Fundamental End-to-End Speech Recognition Toolkitcited by 120
4 more
  1. 2024The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speechcited by 118
  2. 2024Target Speaker Extraction by Directly Exploiting Contextual Information in the Time-Frequency Domaincited by 11
  3. 2023The Multimodal Information Based Speech Processing (MISP) 2023 Challenge: Audio-Visual Target Speaker Extractioncited by 9
  4. 2026Identity-Faithful Audio-Visual Target Speaker Extraction with REAL-2MIX and VOXBLINK2-AVSEcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Separate First, Then Associate: A Two-Stage Approach for Real-World Audio-Visual Speech Enhancement
  2. 2026The ISCSLP 2026 Real-World Audio-Visual Speech Enhancement Challenge: A Benchmark with Natural Mixtures and Degraded Video
  3. 2025From Coarse to Fine: Recursive Audio-Visual Semantic Enhancement for Speech Separation ↗
  4. 2025Audio-Visual Speech Enhancement In Complex Scenarios With Separation And Dereverberation Joint Modeling
  5. 2026DAVE: A Decoupled Audio-Visual Enhancement Framework for Real-World Speech Separation
  6. 2023AV2Wav: Diffusion-Based Re-synthesis from Continuous Self-supervised Features for Audio-Visual Speech Enhancement
  7. 2026Adapting Personalized Speech Enhancement for Low-Latency Audio-Visual Target-Speaker Extraction
  8. 2025Real-Time Audio-Visual Speech Enhancement Using Pre-trained Visual Representations