# Differentiable Consistency Constraints for Improved Deep Speech Enhancement

arXiv: https://arxiv.org/abs/1811.08521 · HTML: https://mlsanity.com/paper/1811.08521/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2018.

## Built on by (most cited first, 30 of 35)

- [HiFi++: a Unified Framework for Bandwidth Extension and Speech Enhancement](https://mlsanity.com/paper/2203.13086/) (2022)
- [AudioScopeV2: Audio-Visual Attention Architectures for Calibrated Open-Domain On-Screen Sound Separation](https://mlsanity.com/paper/2207.10141/) (2022)
- [Real-Time Target Sound Extraction](https://mlsanity.com/paper/2211.02250/) (2022)
- [RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing](https://mlsanity.com/paper/2202.08862/) (2022)
- [Compute and memory efficient universal sound source separation](https://mlsanity.com/paper/2103.02644/) (2021)
- [Heterogeneous Target Speech Separation](https://mlsanity.com/paper/2204.03594/) (2022)
- [Teacher-Student MixIT for Unsupervised and Semi-supervised Speech Separation](https://mlsanity.com/paper/2106.07843/) (2021)
- [Separate but Together: Unsupervised Federated Learning for Speech Enhancement from Non-IID Data](https://mlsanity.com/paper/2105.04727/) (2021)
- [Weight, Block or Unit? Exploring Sparsity Tradeoffs for Speech Enhancement on Tiny Neural Accelerators](https://mlsanity.com/paper/2111.02351/) (2021)
- [Adversarial Permutation Invariant Training for Universal Sound Separation](https://mlsanity.com/paper/2210.12108/) (2022)
- [SCP-GAN: Self-Correcting Discriminator Optimization for Training Consistency Preserving Metric GAN on Speech Enhancement Tasks](https://mlsanity.com/paper/2210.14474/) (2022)
- [End-to-End Neural Speech Coding for Real-Time Communications](https://mlsanity.com/paper/2201.09429/) (2022)
- [MixCycle: Unsupervised Speech Separation via Cyclic Mixture Permutation Invariant Training](https://mlsanity.com/paper/2202.03875/) (2022)
- [Latent-Domain Predictive Neural Speech Coding](https://mlsanity.com/paper/2207.08363/) (2022)
- [Self-Supervised Learning from Automatically Separated Sound Scenes](https://mlsanity.com/paper/2105.02132/) (2021)
- [Continual self-training with bootstrapped remixing for speech enhancement](https://mlsanity.com/paper/2110.10103/) (2021)
- [Improving On-Screen Sound Separation for Open-Domain Videos with Audio-Visual Self-Attention](https://mlsanity.com/paper/2106.09669/) (2021)
- [Optimal Condition Training for Target Source Separation](https://mlsanity.com/paper/2211.05927/) (2022)
- [UNSSOR: Unsupervised Neural Speech Separation by Leveraging Over-determined Training Mixtures](https://mlsanity.com/paper/2305.20054/) (2023)
- [MUSE: Flexible Voiceprint Receptive Fields and Multi-Path Fusion Enhanced Taylor Transformer for U-Net-based Speech Enhancement](https://mlsanity.com/paper/2406.04589/) (2024)
- [Semi-Supervised Object Detection: A Survey on Progress from CNN to Transformer](https://mlsanity.com/paper/2407.08460/) (2024)
- [xLSTM-SENet: xLSTM for Single-Channel Speech Enhancement](https://mlsanity.com/paper/2501.06146/) (2025)
- [Cross-Scale Vector Quantization for Scalable Neural Speech Coding](https://mlsanity.com/paper/2207.03067/) (2022)
- [MedleyVox: An Evaluation Dataset for Multiple Singing Voices Separation](https://mlsanity.com/paper/2211.07302/) (2022)
- [Deep model with built-in cross-attention alignment for acoustic echo cancellation](https://mlsanity.com/paper/2208.11308/) (2022)
- [Towards Sub-millisecond Latency Real-Time Speech Enhancement Models on Hearables](https://mlsanity.com/paper/2409.18239/) (2024)
- [MambAttention: Mamba with Multi-Head Attention for Generalizable Single-Channel Speech Enhancement](https://mlsanity.com/paper/2507.00966/) (2025)
- [Performance optimizations on deep noise suppression models](https://mlsanity.com/paper/2110.04378/) (2021)
- [SANE-TTS: Stable And Natural End-to-End Multilingual Text-to-Speech](https://mlsanity.com/paper/2206.12132/) (2022)
- [Time-Variance Aware Real-Time Speech Enhancement](https://mlsanity.com/paper/2302.13063/) (2023)

## Abstract (arXiv)

In recent years, deep networks have led to dramatic improvements in speech enhancement by framing it as a data-driven pattern recognition problem. In many modern enhancement systems, large amounts of data are used to train a deep network to estimate masks for complex-valued short-time Fourier transforms (STFTs) to suppress noise and preserve speech. However, current masking approaches often neglect two important constraints: STFT consistency and mixture consistency. Without STFT consistency, the system's output is not necessarily the STFT of a time-domain signal, and without mixture consistency, the sum of the estimated sources does not necessarily equal the input mixture. Furthermore, the only previous approaches that apply mixture consistency use real-valued masks; mixture consistency has been ignored for complex-valued masks. In this paper, we show that STFT consistency and mixture consistency can be jointly imposed by adding simple differentiable projection layers to the enhancement network. These layers are compatible with real or complex-valued masks. Using both of these constraints with complex-valued masks provides a 0.7 dB increase in scale-invariant signal-to-distortion ratio (SI-SDR) on a large dataset of speech corrupted by a wide variety of nonstationary noise across a range of input SNRs.