# Look Once to Hear: Target Speech Hearing with Noisy Examples arXiv: https://arxiv.org/abs/2405.06289 ยท HTML: https://mlsanity.com/paper/2405.06289/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Builds on 39 of its 73 references that are in the map. - First posted on arXiv 10 May 2024. ## Builds on (most cited first, 30 of 39) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023) - [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019) - [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018) - [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018) - [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018) - [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017) - [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019) - [Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation](https://mlsanity.com/paper/1804.03619/) (2018) - [WHAM!: Extending Speech Separation to Noisy Environments](https://mlsanity.com/paper/1907.01160/) (2019) - [DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement](https://mlsanity.com/paper/2008.00264/) (2020) - [TasNet: time-domain audio separation network for real-time, single-channel speech separation](https://mlsanity.com/paper/1711.00541/) (2017) - [The Conversation: Deep Audio-Visual Speech Enhancement](https://mlsanity.com/paper/1804.04121/) (2018) - [VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking](https://mlsanity.com/paper/1810.04826/) (2018) - [ICASSP 2023 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2303.11510/) (2023) - [TitaNet: Neural Model for speaker representation with 1D Depth-wise separable convolutions and global context](https://mlsanity.com/paper/2110.04410/) (2021) - [TF-GridNet: Making Time-Frequency Domain Models Great Again for Monaural Speaker Separation](https://mlsanity.com/paper/2209.03952/) (2022) - [Asteroid: the PyTorch-based audio source separation toolkit for researchers](https://mlsanity.com/paper/2005.04132/) (2020) - [DeepFilterNet: A Low Complexity Speech Enhancement Framework for Full-Band Audio based on Deep Filtering](https://mlsanity.com/paper/2110.05588/) (2021) - [SpEx+: A Complete Time Domain Speaker Extraction Network](https://mlsanity.com/paper/2005.04686/) (2020) - [Neural Target Speech Extraction: An Overview](https://mlsanity.com/paper/2301.13341/) (2023) - [My lips are concealed: Audio-visual speech enhancement through obstructions](https://mlsanity.com/paper/1907.04975/) (2019) - [A Fully Convolutional Neural Network for Speech Enhancement](https://mlsanity.com/paper/1609.07132/) (2016) - [Real-Time Target Sound Extraction](https://mlsanity.com/paper/2211.02250/) (2022) - [The Cone of Silence: Speech Separation by Localization](https://mlsanity.com/paper/2010.06007/) (2020) - [Personalized Speech Enhancement: New Models and Comprehensive Evaluation](https://mlsanity.com/paper/2110.09625/) (2021) - [Multi-modal Multi-channel Target Speech Separation](https://mlsanity.com/paper/2003.07032/) (2020) - [Face Landmark-based Speaker-Independent Audio-Visual Speech Enhancement in Multi-Talker Environments](https://mlsanity.com/paper/1811.02480/) (2018) - [Personalized PercepNet: Real-time, Low-complexity Target Voice Separation and Enhancement](https://mlsanity.com/paper/2106.04129/) (2021) - [Real-time binaural speech separation with preserved spatial cues](https://mlsanity.com/paper/2002.06637/) (2020) ## Built on by (most cited first, 11 of 11) - [Knowledge boosting during low-latency inference](https://mlsanity.com/paper/2407.11055/) (2024) - [Target conversation extraction: Source separation using turn-taking dynamics](https://mlsanity.com/paper/2407.11277/) (2024) - [TS-SUPERB: A Target Speech Processing Benchmark for Speech Self-Supervised Learning Models](https://mlsanity.com/paper/2505.06660/) (2025) - [Spatial Speech Translation: Translating Across Space With Binaural Hearables](https://mlsanity.com/paper/2504.18715/) (2025) - [Neural Speech Extraction with Human Feedback](https://mlsanity.com/paper/2508.03041/) (2025) - [Target Speaker Extraction through Comparing Noisy Positive and Negative Audio Enrollments](https://mlsanity.com/paper/2502.16611/) (2025) - [SonicSieve: Bringing Directional Speech Extraction to Smartphones Using Acoustic Microstructures](https://mlsanity.com/paper/2504.10793/) (2025) - [TF-MLPNet: Tiny Real-Time Neural Speech Separation](https://mlsanity.com/paper/2508.03047/) (2025) - [Proactive Hearing Assistants that Isolate Egocentric Conversations](https://mlsanity.com/paper/2511.11473/) (2025) - [MoXaRt: Audio-Visual Object-Guided Sound Interaction for XR](https://mlsanity.com/paper/2603.10465/) (2026) - [Fine-grained Soundscape Control for Augmented Hearing](https://mlsanity.com/paper/2603.00395/) (2026) ## Abstract (arXiv) In crowded settings, the human brain can focus on speech from a target speaker, given prior knowledge of how they sound. We introduce a novel intelligent hearable system that achieves this capability, enabling target speech hearing to ignore all interfering speech and noise, but the target speaker. A naive approach is to require a clean speech example to enroll the target speaker. This is however not well aligned with the hearable application domain since obtaining a clean example is challenging in real world scenarios, creating a unique user interface problem. We present the first enrollment interface where the wearer looks at the target speaker for a few seconds to capture a single, short, highly noisy, binaural example of the target speaker. This noisy example is used for enrollment and subsequent speech extraction in the presence of interfering speakers and noise. Our system achieves a signal quality improvement of 7.01 dB using less than 5 seconds of noisy enrollment audio and can process 8 ms of audio chunks in 6.24 ms on an embedded CPU. Our user studies demonstrate generalization to real-world static and mobile speakers in previously unseen indoor and outdoor multipath environments. Finally, our enrollment interface for noisy examples does not cause performance degradation compared to clean examples, while being convenient and user-friendly. Taking a step back, this paper takes an important step towards enhancing the human auditory perception with artificial intelligence. We provide code and data at: https://github.com/vb000/LookOnceToHear.