mlsanityData to 19 Oct 2026

UltraM2M: Leveraging Text Transcripts and Mixture Constraints for Weakly-Supervised Speech Enhancement

arXiv 2610.05155 ↗First posted 4 Oct 2026eess.AScited by 0

On the map: Speech and audio language models → Speech enhancement and noise suppression

Abstract (arXiv)

We propose UltraM2M, a weakly-supervised speech enhancement algorithm building upon the recent unsupervised mixture-to-mixture (M2M) algorithm. M2M realizes unsupervised speech enhancement by training deep neural networks on a set of real-recorded noisy-reverberant multi-channel mixture signals to estimate target speech and non-target signals. The two estimated signals are penalized by a so-called mixture-constraint (MC) loss, which constrains them to reconstruct the observed mixture signals. Although shown to be effective, the mixture constraint may be too weak to enable sufficient noise reduction. To deal with this, UltraM2M extends M2M by further leveraging text transcripts of real-recorded mixtures to design an automatic speech recognition (ASR) loss to penalize the estimated speech signal. The ASR loss can be viewed as a form of weak supervision that could help unsupervised enhancement. Evaluation results on the CHiME-4 dataset show the effectiveness of UltraM2M.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models. Builds on 27 of its 36 references that are in the map. First posted on arXiv 4 Oct 2026.

Builds on

27 of 36 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Advances in Microphone Array Processing and Multichannel Speech Enhancement13 Feb
  2. 2025Reducing the Gap Between Pretrained Speech Enhancement and Recognition Models Using a Real Speech-Trained Bridging Module5 Jan
  3. 2024SuperM2M: Supervised and Mixture-to-Mixture Co-Learning for Speech Enhancement and Noise-Robust ASR15 Mar
  4. 2023D4AM: A General Denoising Framework for Downstream Acoustic Models28 Nov
  5. 2023How does end-to-end speech recognition training impact speech enhancement artifacts?20 Nov
  6. 2023Unsupervised speech enhancement with diffusion-based generative models19 Sep
  7. 2023The CHiME-7 UDASE task: Unsupervised domain adaptation for conversational speech enhancement7 Jul
  8. 2023Gradient Remedy for Multi-Task Learning in End-to-End Noise-Robust Speech Recognition22 Feb
  9. 2022On monoaural speech enhancement for automatic recognition of real noisy speech using mixture invariant training3 May
  10. 2022End-to-End Integration of Speech Recognition, Speech Enhancement, and Self-Supervised Learning Representation1 Apr
  11. 2022How Bad Are Artifacts?: Analyzing the Impact of Speech Enhancement Errors on ASR18 Jan
  12. 2021Closing the Gap Between Time-Domain Multi-Channel Speech Enhancement on Real and Simulation Conditions27 Oct
1 more
  1. 2021Human Listening and Live Captioning: Multi-Task Training for Speech Enhancement5 Jun

Other references · 14

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 4,385
  2. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 2,135
  3. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 754
  4. 2021DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressorscited by 292
  5. 2018The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselinescited by 154
  6. 2022TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separationcited by 121
  7. 2019MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognitioncited by 67
  8. 2021Convolutive Prediction for Monaural Speech Dereverberation and Noisy-Reverberant Speaker Separationcited by 21
  9. 2019Finding Strength in Weakness: Learning to Separate Sounds with Weak Supervisioncited by 18
  10. 2022End-to-End Integration of Speech Recognition, Dereverberation, Beamforming, and Self-Supervised Learning Representationcited by 18
  11. 2020Gated Recurrent Fusion with Joint Training Framework for Robust End-to-End Speech Recognitioncited by 17
  12. 2025ArrayDPS: Unsupervised Blind Speech Separation with a Diffusion Priorcited by 10
2 more
  1. 2024Mixture to Mixture: Leveraging Close-talk Mixtures as Weak-supervision for Speech Separationcited by 5
  2. 2026Exploiting Noise Inseparability for Weakly-Supervised Discriminative Speech Denoising Using Noisy Targetscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Incorporating Real-world Noisy Speech in Neural-network-based Speech Enhancement Systems
  2. 2021Unsupervised Speech Enhancement with speech recognition embedding and disentanglement losses
  3. 2025Mixture to Beamformed Mixture: Leveraging Beamformed Mixture as Weak-Supervision for Speech Enhancement and Noise-Robust ASR
  4. 2023Single-channel speech enhancement using learnable loss mixup
  5. 2021MetricGAN-U: Unsupervised speech enhancement/ dereverberation based only on noisy/ reverberated speech
  6. 2026Unsupervised Speech Enhancement via Drifting
  7. 2023Toward Universal Speech Enhancement for Diverse Input Conditions
  8. 2023Unifying Speech Enhancement and Separation with Gradient Modulation for End-to-End Noise-Robust Speech Separation