mlsanityData to 19 Oct 2026

The ISCSLP 2026 Real-World Audio-Visual Speech Enhancement Challenge: A Benchmark with Natural Mixtures and Degraded Video

arXiv 2608.23759 ↗First posted 24 Aug 2026eess.AS cs.SDcited by 0

On the map: Speech and audio language models → Speech enhancement and noise suppression

Abstract (arXiv)

Audio-visual speech enhancement (AVSE) uses a target speaker's visible articulation to recover that speaker's voice from overlapped speech. Yet most evaluation protocols rely on synthetic mixtures and reliable video. The ISCSLP 2026 Real-World AVSE Challenge addresses both gaps. Track~1 combines naturally recorded two-talker mixtures, which lack a clean reference, with reference-available remixes of the same speakers; Track~2 additionally degrades the target video in five ways and adds 3-m far-field recordings. Sixteen and twelve teams were ranked on speaker-disjoint test data by rank averaging over waveform fidelity, predicted quality, transcription accuracy, and speaker similarity. On Track~1 remixes, the best system reaches 12.7~dB SI-SDR and 0.85 STOI, but natural recordings remain harder: even the lowest CER rises from 9.0% to 14.7%. The leading systems use video mainly for speaker attribution rather than signal reconstruction, and the top two lose under 0.5~dB SI-SDR on Track~2. UTMOS and DNSMOS rank systems differently from the other metrics, so no single metric captures target-speech recovery. We release the baselines, evaluator, and official results.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models. Builds on 29 of its 35 references that are in the map. First posted on arXiv 24 Aug 2026.

Builds on

29 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Robust Audio-Visual Speech Enhancement: Correcting Misassignments in Complex Environments with Advanced Post-Processing22 Sep

Other references · 28

  1. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 2,135
  2. 2018VoxCeleb2: Deep Speaker Recognitioncited by 1,166
  3. 2018SDR - half-baked or well done?cited by 626
  4. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 596
  5. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 319
  6. 2021DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressorscited by 292
  7. 2022Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkitcited by 179
  8. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 136
  9. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 127
  10. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 127
  11. 2022TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separationcited by 121
  12. 2024The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speechcited by 118
16 more
  1. 2019Time Domain Audio Visual Speech Separationcited by 33
  2. 2019Audio-visual Speech Enhancement Using Conditional Variational Auto-Encoderscited by 32
  3. 2021Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separationcited by 31
  4. 2025Fun-ASR Technical Reportcited by 28
  5. 2020Improved Lite Audio-Visual Speech Enhancementcited by 19
  6. 2024TIGER: Time-frequency Interleaved Gain Extraction and Reconstruction for Efficient Speech Separationcited by 16
  7. 2023IIANet: An Intra- and Inter-Modality Attention Network for Audio-Visual Speech Separationcited by 14
  8. 2022An Audio-Visual Speech Separation Model Inspired by Cortico-Thalamo-Cortical Circuitscited by 12
  9. 2023The Multimodal Information Based Speech Processing (MISP) 2023 Challenge: Audio-Visual Target Speaker Extractioncited by 9
  10. 2025Advances in Speech Separation: Techniques, Challenges, and Future Trendscited by 5
  11. 2023RTFS-Net: Recurrent Time-Frequency Modelling for Efficient Audio-Visual Speech Separationcited by 4
  12. 2025MeMo: Attentional Momentum for Real-Time Audio-Visual Target Speaker Extraction Under Impaired Visual Conditionscited by 4
  13. 2024AV-CrossNet: an Audiovisual Complex Spectral Mapping Network for Speech Separation By Leveraging Narrow- and Cross-Band Modelingcited by 3
  14. 2024RAVSS: Robust Audio-Visual Speech Separation in Multi-Speaker Scenarios with Missing Visual Cuescited by 3
  15. 2024MoMuSE: Momentum Multi-modal Target Speaker Extraction for Real-time Scenarios with Impaired Visual Cuescited by 3
  16. 2026SLT 2026 REAL-TSE Challenge: Real-world Target Speaker Extraction from Conversational Recordingscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CF-AVSS: A Coarse-to-Fine Audio-Visual Speech Separation Framework with Target-Speaker Refinement and Post-Fusion for Real-World AVSE Challenge
  2. 2026Separate First, Then Associate: A Two-Stage Approach for Real-World Audio-Visual Speech Enhancement
  3. 2025Audio-Visual Speech Enhancement In Complex Scenarios With Separation And Dereverberation Joint Modeling
  4. 2025Real-Time Audio-Visual Speech Enhancement Using Pre-trained Visual Representations
  5. 2026Adapting Personalized Speech Enhancement for Low-Latency Audio-Visual Target-Speaker Extraction
  6. 2023AV2Wav: Diffusion-Based Re-synthesis from Continuous Self-supervised Features for Audio-Visual Speech Enhancement
  7. 2026DAVE: A Decoupled Audio-Visual Enhancement Framework for Real-World Speech Separation
  8. 2022Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis