# ConceptBeam: Concept Driven Target Speech Extraction arXiv: https://arxiv.org/abs/2207.11964 ยท HTML: https://mlsanity.com/paper/2207.11964/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 37 of its 53 references that are in the map. - First posted on arXiv 25 Jul 2022. ## Builds on (most cited first, 30 of 37) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015) - [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018) - [End-to-End Learning of Visual Representations from Uncurated Instructional Videos](https://mlsanity.com/paper/1912.06430/) (2019) - [The Sound of Pixels](https://mlsanity.com/paper/1804.03160/) (2018) - [Deep clustering: Discriminative embeddings for segmentation and separation](https://mlsanity.com/paper/1508.04306/) (2015) - [The Sound of Motions](https://mlsanity.com/paper/1904.05979/) (2019) - [Music Gesture for Visual Sound Separation](https://mlsanity.com/paper/2004.09476/) (2020) - [AVLnet: Learning Audio-Visual Language Representations from Instructional Videos](https://mlsanity.com/paper/2006.09199/) (2020) - [Co-Separating Sounds of Visual Objects](https://mlsanity.com/paper/1904.07750/) (2019) - [The Conversation: Deep Audio-Visual Speech Enhancement](https://mlsanity.com/paper/1804.04121/) (2018) - [Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networks](https://mlsanity.com/paper/1703.06284/) (2017) - [Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input](https://mlsanity.com/paper/1804.01452/) (2018) - [Recursive Visual Sound Separation Using Minus-Plus Net](https://mlsanity.com/paper/1908.11602/) (2019) - [Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptions](https://mlsanity.com/paper/2105.04489/) (2021) - [ESPnet-SLU: Advancing Spoken Language Understanding through ESPnet](https://mlsanity.com/paper/2111.14706/) (2021) - [Text-Free Image-to-Speech Synthesis Using Learned Segmental Units](https://mlsanity.com/paper/2012.15454/) (2020) - [Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech](https://mlsanity.com/paper/1911.09602/) (2019) - [Word Discovery in Visually Grounded, Self-Supervised Speech Models](https://mlsanity.com/paper/2203.15081/) (2022) - [Large-scale representation learning from visually grounded untranscribed speech](https://mlsanity.com/paper/1909.08782/) (2019) - [Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques](https://mlsanity.com/paper/2104.13225/) (2021) - [Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling](https://mlsanity.com/paper/2202.03543/) (2022) - [Fast-Slow Transformer for Visually Grounding Speech](https://mlsanity.com/paper/2109.08186/) (2021) - [What's All the FUSS About Free Universal Sound Separation Data?](https://mlsanity.com/paper/2011.00803/) (2020) - [Language learning using Speech to Image retrieval](https://mlsanity.com/paper/1909.03795/) (2019) - [Talk, Don't Write: A Study of Direct Speech-Based Image Retrieval](https://mlsanity.com/paper/2104.01894/) (2021) - [SoundBeam: Target sound extraction conditioned on sound-class labels and enrollment clues for increased performance and continuous learning](https://mlsanity.com/paper/2204.03895/) (2022) - [Attention-Based Keyword Localisation in Speech using Visual Grounding](https://mlsanity.com/paper/2106.08859/) (2021) - [Heterogeneous Target Speech Separation](https://mlsanity.com/paper/2204.03594/) (2022) ## Built on by (most cited first, 8 of 8) - [SoloSpeech: Enhancing Intelligibility and Quality in Target Speech Extraction through a Cascaded Generative Pipeline](https://mlsanity.com/paper/2505.19314/) (2025) - [Typing to Listen at the Cocktail Party: Text-Guided Target Speaker Extraction](https://mlsanity.com/paper/2310.07284/) (2023) - [Optimal Condition Training for Target Source Separation](https://mlsanity.com/paper/2211.05927/) (2022) - [Target conversation extraction: Source separation using turn-taking dynamics](https://mlsanity.com/paper/2407.11277/) (2024) - [Multichannel-to-Multichannel Target Sound Extraction Using Direction and Timestamp Clues](https://mlsanity.com/paper/2409.12415/) (2024) - [Contextual Speech Extraction: Leveraging Textual History as an Implicit Cue for Target Speech Extraction](https://mlsanity.com/paper/2503.08798/) (2025) - [Neural Speech Extraction with Human Feedback](https://mlsanity.com/paper/2508.03041/) (2025) - [SoundCompass: Navigating Target Sound Extraction With Effective Directional Clue Integration In Complex Acoustic Scenes](https://mlsanity.com/paper/2509.18561/) (2025) ## Abstract (arXiv) We propose a novel framework for target speech extraction based on semantic information, called ConceptBeam. Target speech extraction means extracting the speech of a target speaker in a mixture. Typical approaches have been exploiting properties of audio signals, such as harmonic structure and direction of arrival. In contrast, ConceptBeam tackles the problem with semantic clues. Specifically, we extract the speech of speakers speaking about a concept, i.e., a topic of interest, using a concept specifier such as an image or speech. Solving this novel problem would open the door to innovative applications such as listening systems that focus on a particular topic discussed in a conversation. Unlike keywords, concepts are abstract notions, making it challenging to directly represent a target concept. In our scheme, a concept is encoded as a semantic embedding by mapping the concept specifier to a shared embedding space. This modality-independent space can be built by means of deep metric learning using paired data consisting of images and their spoken captions. We use it to bridge modality-dependent information, i.e., the speech segments in the mixture, and the specified, modality-independent concept. As a proof of our scheme, we performed experiments using a set of images associated with spoken captions. That is, we generated speech mixtures from these spoken captions and used the images or speech signals as the concept specifiers. We then extracted the target speech using the acoustic characteristics of the identified segments. We compare ConceptBeam with two methods: one based on keywords obtained from recognition systems and another based on sound source separation. We show that ConceptBeam clearly outperforms the baseline methods and effectively extracts speech based on the semantic representation.