# Don't Listen to Me: A Lightweight, Low-Latency Model for Own-Voice Cancellation in Far-Field Speech Enhancement

arXiv: https://arxiv.org/abs/2606.23332 · HTML: https://mlsanity.com/paper/2606.23332/

## Key facts

- No citing papers in the map yet (data through 20 Oct 2026).
- Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models.
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 22 Jun 2026.

## Placed in

- Speech and audio language models / [Speech enhancement and noise suppression](https://mlsanity.com/speech/speech-enhancement-and-noise-suppression/)

## Builds on (most cited first, 22 of 22)

- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023)
- [Diagonal State Spaces are as Effective as Structured State Spaces](https://mlsanity.com/paper/2203.14343/) (2022)
- [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019)
- [WHAM!: Extending Speech Separation to Noisy Environments](https://mlsanity.com/paper/1907.01160/) (2019)
- [LibriMix: An Open-Source Dataset for Generalizable Speech Separation](https://mlsanity.com/paper/2005.11262/) (2020)
- [TasNet: time-domain audio separation network for real-time, single-channel speech separation](https://mlsanity.com/paper/1711.00541/) (2017)
- [Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers](https://mlsanity.com/paper/2407.09941/) (2024)
- [Improving speaker discrimination of target speech extraction with time-domain SpeakerBeam](https://mlsanity.com/paper/2001.08378/) (2020)
- [Were RNNs All We Needed?](https://mlsanity.com/paper/2410.01201/) (2024)
- [Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs](https://mlsanity.com/paper/2502.15938/) (2025)
- [Distillation and Pruning for Scalable Self-Supervised Representation-Based Speech Quality Assessment](https://mlsanity.com/paper/2502.05356/) (2025)
- [TF-Locoformer: Transformer with Local Modeling by Convolution for Speech Separation and Enhancement](https://mlsanity.com/paper/2408.03440/) (2024)
- [Separate and Reconstruct: Asymmetric Encoder-Decoder for Speech Separation](https://mlsanity.com/paper/2406.05983/) (2024)
- [ICASSP 2023 Acoustic Echo Cancellation Challenge](https://mlsanity.com/paper/2309.12553/) (2023)
- [SpeakerBeam-SS: Real-time Target Speaker Extraction with Lightweight Conv-TasNet and State Space Modeling](https://mlsanity.com/paper/2407.01857/) (2024)
- [Listen only to me! How well can target speech extraction handle false alarms?](https://mlsanity.com/paper/2204.04811/) (2022)
- [Target Speech Extraction with Pre-trained Self-supervised Learning Models](https://mlsanity.com/paper/2402.13199/) (2024)
- [SepMamba: State-space models for speaker separation using Mamba](https://mlsanity.com/paper/2410.20997/) (2024)

## Abstract (arXiv)

We introduce own-voice cancellation (OVC): removing a target (enrolled) speaker from a noisy multi-speaker mixture while preserving any remaining speech. Framed as the complement of target speaker extraction, OVC addresses latency-induced own-voice artifacts that arise when a far-field device streams enhanced audio back to the user, as the round-trip time easily exceeds the perceptual threshold for own-voice distortion. We condition a time-domain model with only 2 ms algorithmic latency on a short enrollment utterance and benchmark TD-SpeakerBeam alongside a lighter Mamba-MinGRU masker built from Mamba blocks with MinGRU temporal mixing. Replacing the ConvTasNet-based auxiliary network with a linear RNN encoder improves both signal-to-distortion ratio and predicted MOS while reducing compute. Results establish OVC as a practical, low-latency enhancement objective for far-field denoising.