# Target conversation extraction: Source separation using turn-taking dynamics

arXiv: https://arxiv.org/abs/2407.11277 · HTML: https://mlsanity.com/paper/2407.11277/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 38 references that are in the map.
- First posted on arXiv 15 Jul 2024.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselines](https://mlsanity.com/paper/2010.11567/) (2020)
- [Speaker Diarization with LSTM](https://mlsanity.com/paper/1710.10468/) (2017)
- [TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separation](https://mlsanity.com/paper/2211.12433/) (2022)
- [Poolingformer: Long Document Modeling with Pooling Attention](https://mlsanity.com/paper/2105.04371/) (2021)
- [Voice Activity Projection: Self-supervised Learning of Turn-taking Events](https://mlsanity.com/paper/2205.09812/) (2022)
- [Separate Anything You Describe](https://mlsanity.com/paper/2308.05037/) (2023)
- [Open Source MagicData-RAMC: A Rich Annotated Mandarin Conversational(RAMC) Speech Dataset](https://mlsanity.com/paper/2203.16844/) (2022)
- [Streaming Multi-Talker ASR with Token-Level Serialized Output Training](https://mlsanity.com/paper/2202.00842/) (2022)
- [End-to-end Spoken Conversational Question Answering: Task, Dataset and Model](https://mlsanity.com/paper/2204.14272/) (2022)
- [Neural Target Speech Extraction: An Overview](https://mlsanity.com/paper/2301.13341/) (2023)
- [Multilogue-Net: A Context Aware RNN for Multi-modal Emotion Detection and Sentiment Analysis in Conversation](https://mlsanity.com/paper/2002.08267/) (2020)
- [Speech Recognition and Multi-Speaker Diarization of Long Conversations](https://mlsanity.com/paper/2005.08072/) (2020)
- [Real-Time Target Sound Extraction](https://mlsanity.com/paper/2211.02250/) (2022)
- [Real-time and Continuous Turn-taking Prediction Using Voice Activity Projection](https://mlsanity.com/paper/2401.04868/) (2024)
- [From Simulated Mixtures to Simulated Conversations as Training Data for End-to-End Neural Diarization](https://mlsanity.com/paper/2204.00890/) (2022)
- [AV-SepFormer: Cross-Attention SepFormer for Audio-Visual Target Speaker Extraction](https://mlsanity.com/paper/2306.14170/) (2023)
- [Speech-Text Dialog Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignment](https://mlsanity.com/paper/2305.11579/) (2023)
- [Resource-Efficient Separation Transformer](https://mlsanity.com/paper/2206.09507/) (2022)
- [Look Once to Hear: Target Speech Hearing with Noisy Examples](https://mlsanity.com/paper/2405.06289/) (2024)
- [Simulating realistic speech overlaps improves multi-talker ASR](https://mlsanity.com/paper/2210.15715/) (2022)
- [Hybrid Neural Networks for On-device Directional Hearing](https://mlsanity.com/paper/2112.05893/) (2021)
- [Conversational Speech Recognition By Learning Conversation-level Characteristics](https://mlsanity.com/paper/2202.07855/) (2022)
- [ConceptBeam: Concept Driven Target Speech Extraction](https://mlsanity.com/paper/2207.11964/) (2022)
- [Automatic Evaluation of Turn-taking Cues in Conversational Speech Synthesis](https://mlsanity.com/paper/2305.17971/) (2023)
- [Efficient Transformer for Direct Speech Translation](https://mlsanity.com/paper/2107.03069/) (2021)

## Built on by (most cited first, 4 of 4)

- [Spatial Speech Translation: Translating Across Space With Binaural Hearables](https://mlsanity.com/paper/2504.18715/) (2025)
- [Visual Cues Support Robust Turn-taking Prediction in Noise](https://mlsanity.com/paper/2505.22088/) (2025)
- [Target Speaker Extraction through Comparing Noisy Positive and Negative Audio Enrollments](https://mlsanity.com/paper/2502.16611/) (2025)
- [Proactive Hearing Assistants that Isolate Egocentric Conversations](https://mlsanity.com/paper/2511.11473/) (2025)

## Abstract (arXiv)

Extracting the speech of participants in a conversation amidst interfering speakers and noise presents a challenging problem. In this paper, we introduce the novel task of target conversation extraction, where the goal is to extract the audio of a target conversation based on the speaker embedding of one of its participants. To accomplish this, we propose leveraging temporal patterns inherent in human conversations, particularly turn-taking dynamics, which uniquely characterize speakers engaged in conversation and distinguish them from interfering speakers and noise. Using neural networks, we show the feasibility of our approach on English and Mandarin conversation datasets. In the presence of interfering speakers, our results show an 8.19 dB improvement in signal-to-noise ratio for 2-speaker conversations and a 7.92 dB improvement for 2-4-speaker conversations. Code, dataset available at https://github.com/chentuochao/Target-Conversation-Extraction.