# SuperM2M: Supervised and Mixture-to-Mixture Co-Learning for Speech Enhancement and Noise-Robust ASR arXiv: https://arxiv.org/abs/2403.10271 ยท HTML: https://mlsanity.com/paper/2403.10271/ ## Key facts - Cited by 6 papers in the map (data through 19 Oct 2026). - Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models. - Builds on 56 of its 88 references that are in the map. - First posted on arXiv 15 Mar 2024. ## Placed in - Speech and audio language models / [Speech enhancement and noise suppression](https://mlsanity.com/speech/speech-enhancement-and-noise-suppression/) ## Builds on (most cited first, 30 of 56) - [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022) - [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021) - [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018) - [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018) - [Supervised Speech Separation Based on Deep Learning: An Overview](https://mlsanity.com/paper/1708.07524/) (2017) - [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019) - [CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings](https://mlsanity.com/paper/2004.09249/) (2020) - [Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation](https://mlsanity.com/paper/1804.03619/) (2018) - [The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselines](https://mlsanity.com/paper/1803.10609/) (2018) - [M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge](https://mlsanity.com/paper/2110.07393/) (2021) - [Continuous speech separation: dataset and analysis](https://mlsanity.com/paper/2001.11482/) (2020) - [TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separation](https://mlsanity.com/paper/2211.12433/) (2022) - [Wavesplit: End-to-End Speech Separation by Speaker Clustering](https://mlsanity.com/paper/2002.08933/) (2020) - [MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognition](https://mlsanity.com/paper/1910.06522/) (2019) - [Voice Separation with an Unknown Number of Multiple Speakers](https://mlsanity.com/paper/2003.01531/) (2020) - [Differentiable Consistency Constraints for Improved Deep Speech Enhancement](https://mlsanity.com/paper/1811.08521/) (2018) - [SpEx: Multi-Scale Time Domain Speaker Extraction Network](https://mlsanity.com/paper/2004.08326/) (2020) - [Neural Target Speech Extraction: An Overview](https://mlsanity.com/paper/2301.13341/) (2023) - [The CHiME-7 DASR Challenge: Distant Meeting Transcription with Multiple Devices in Diverse Scenarios](https://mlsanity.com/paper/2306.13734/) (2023) - [Universal Sound Separation](https://mlsanity.com/paper/1905.03330/) (2019) - [Far-Field Automatic Speech Recognition](https://mlsanity.com/paper/2009.09395/) (2020) - [End-to-End Integration of Speech Recognition, Speech Enhancement, and Self-Supervised Learning Representation](https://mlsanity.com/paper/2204.00540/) (2022) - [RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing](https://mlsanity.com/paper/2202.08862/) (2022) - [SpatialNet: Extensively Learning Spatial Information for Multichannel Joint Speech Separation, Denoising and Dereverberation](https://mlsanity.com/paper/2307.16516/) (2023) - [Multi-microphone Complex Spectral Mapping for Utterance-wise and Continuous Speech Separation](https://mlsanity.com/paper/2010.01703/) (2020) - [On The Compensation Between Magnitude and Phase in Speech Separation](https://mlsanity.com/paper/2108.05470/) (2021) - [A Noise-Robust Self-supervised Pre-training Model Based Speech Representation Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2201.08930/) (2022) - [Toward Universal Speech Enhancement for Diverse Input Conditions](https://mlsanity.com/paper/2309.17384/) (2023) - [Noisy-target Training: A Training Strategy for DNN-based Speech Enhancement without Clean Speech](https://mlsanity.com/paper/2101.08625/) (2021) - [AudioScopeV2: Audio-Visual Attention Architectures for Calibrated Open-Domain On-Screen Sound Separation](https://mlsanity.com/paper/2207.10141/) (2022) ## Built on by (most cited first, 6 of 6) - [ctPuLSE: Close-Talk, and Pseudo-Label Based Far-Field, Speech Enhancement](https://mlsanity.com/paper/2407.19485/) (2024) - [SuPseudo: A Pseudo-supervised Learning Method for Neural Speech Enhancement in Far-field Speech Recognition](https://mlsanity.com/paper/2505.24450/) (2025) - [DeCodec: Rethinking Audio Codecs as Universal Disentangled Representation Learners](https://mlsanity.com/paper/2509.09201/) (2025) - [Mixture to Beamformed Mixture: Leveraging Beamformed Mixture as Weak-Supervision for Speech Enhancement and Noise-Robust ASR](https://mlsanity.com/paper/2507.15229/) (2025) - [Generating Training Targets for Real-World Speech Enhancement via Close-to-Distant Microphone Projection](https://mlsanity.com/paper/2606.13109/) (2026) - [UltraM2M: Leveraging Text Transcripts and Mixture Constraints for Weakly-Supervised Speech Enhancement](https://mlsanity.com/paper/2610.05155/) (2026) ## Abstract (arXiv) The current dominant approach for neural speech enhancement is based on supervised learning by using simulated training data. The trained models, however, often exhibit limited generalizability to real-recorded data. To address this, this paper investigates training enhancement models directly on real target-domain data. We propose to adapt mixture-to-mixture (M2M) training, originally designed for speaker separation, for speech enhancement, by modeling multi-source noise signals as a single, combined source. In addition, we propose a co-learning algorithm that improves M2M with the help of supervised algorithms. When paired close-talk and far-field mixtures are available for training, M2M realizes speech enhancement by training a deep neural network (DNN) to produce speech and noise estimates in a way such that they can be linearly filtered to reconstruct the close-talk and far-field mixtures. This way, the DNN can be trained directly on real mixtures, and can leverage close-talk and far-field mixtures as a weak supervision to enhance far-field mixtures. To improve M2M, we combine it with supervised approaches to co-train the DNN, where mini-batches of real close-talk and far-field mixture pairs and mini-batches of simulated mixture and clean speech pairs are alternately fed to the DNN, and the loss functions are respectively (a) the mixture reconstruction loss on the real close-talk and far-field mixtures and (b) the regular enhancement loss on the simulated clean speech and noise. We find that, this way, the DNN can learn from real and simulated data to achieve better generalization to real data. We name this algorithm SuperM2M (supervised and mixture-to-mixture co-learning). Evaluation results on the CHiME-4 dataset show its effectiveness and potential.