# Neural Speech Extraction with Human Feedback

arXiv: https://arxiv.org/abs/2508.03041 · HTML: https://mlsanity.com/paper/2508.03041/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 5 Aug 2025.

## Builds on (most cited first, 24 of 24)

- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Attention is All You Need in Speech Separation](https://mlsanity.com/paper/2010.13154/) (2020)
- [DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors](https://mlsanity.com/paper/2110.01763/) (2021)
- [WHAM!: Extending Speech Separation to Noisy Environments](https://mlsanity.com/paper/1907.01160/) (2019)
- [EditGAN: High-Precision Semantic Image Editing](https://mlsanity.com/paper/2111.03186/) (2021)
- [Speaker Diarization with LSTM](https://mlsanity.com/paper/1710.10468/) (2017)
- [TF-GridNet: Making Time-Frequency Domain Models Great Again for Monaural Speaker Separation](https://mlsanity.com/paper/2209.03952/) (2022)
- [Separate Anything You Describe](https://mlsanity.com/paper/2308.05037/) (2023)
- [Neural Target Speech Extraction: An Overview](https://mlsanity.com/paper/2301.13341/) (2023)
- [MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Models](https://mlsanity.com/paper/2402.06178/) (2024)
- [Real-Time Target Sound Extraction](https://mlsanity.com/paper/2211.02250/) (2022)
- [Prompt-guided Precise Audio Editing with Diffusion Models](https://mlsanity.com/paper/2406.04350/) (2024)
- [Don't shoot butterfly with rifles: Multi-channel Continuous Speech Separation with Early Exit Transformer](https://mlsanity.com/paper/2010.12180/) (2020)
- [AV-SepFormer: Cross-Attention SepFormer for Audio-Visual Target Speaker Extraction](https://mlsanity.com/paper/2306.14170/) (2023)
- [Resource-Efficient Separation Transformer](https://mlsanity.com/paper/2206.09507/) (2022)
- [Look Once to Hear: Target Speech Hearing with Noisy Examples](https://mlsanity.com/paper/2405.06289/) (2024)
- [MusiConGen: Rhythm and Chord Control for Transformer-Based Text-to-Music Generation](https://mlsanity.com/paper/2407.15060/) (2024)
- [Hybrid Neural Networks for On-device Directional Hearing](https://mlsanity.com/paper/2112.05893/) (2021)
- [ConceptBeam: Concept Driven Target Speech Extraction](https://mlsanity.com/paper/2207.11964/) (2022)
- [Latent Iterative Refinement for Modular Source Separation](https://mlsanity.com/paper/2211.11917/) (2022)
- [Fine-Grained and Interpretable Neural Speech Editing](https://mlsanity.com/paper/2407.05471/) (2024)

## Built on by (most cited first, 2 of 2)

- [GenTSE: Enhancing Target Speaker Extraction via a Coarse-to-Fine Generative Language Model](https://mlsanity.com/paper/2512.20978/) (2025)
- [Fine-grained Soundscape Control for Augmented Hearing](https://mlsanity.com/paper/2603.00395/) (2026)

## Abstract (arXiv)

We present the first neural target speech extraction (TSE) system that uses human feedback for iterative refinement. Our approach allows users to mark specific segments of the TSE output, generating an edit mask. The refinement system then improves the marked sections while preserving unmarked regions. Since large-scale datasets of human-marked errors are difficult to collect, we generate synthetic datasets using various automated masking functions and train models on each. Evaluations show that models trained with noise power-based masking (in dBFS) and probabilistic thresholding perform best, aligning with human annotations. In a study with 22 participants, users showed a preference for refined outputs over baseline TSE. Our findings demonstrate that human-in-the-loop refinement is a promising approach for improving the performance of neural speech extraction.