# VoiceFilter-Lite: Streaming Targeted Voice Separation for On-Device Speech Recognition

arXiv: https://arxiv.org/abs/2009.04323 · HTML: https://mlsanity.com/paper/2009.04323/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Sep 2020.

## Built on by (most cited first, 30 of 32)

- [Separate What You Describe: Language-Queried Audio Source Separation](https://mlsanity.com/paper/2203.15147/) (2022)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [Personalized Speech Enhancement: New Models and Comprehensive Evaluation](https://mlsanity.com/paper/2110.09625/) (2021)
- [AdaptiveNet: Post-deployment Neural Architecture Adaptation for Diverse Edge Environments](https://mlsanity.com/paper/2303.07129/) (2023)
- [Towards Neural Diarization for Unlimited Numbers of Speakers Using Global and Local Attractors](https://mlsanity.com/paper/2107.01545/) (2021)
- [Fast Real-time Personalized Speech Enhancement: End-to-End Enhancement Network (E3Net) and Knowledge Distillation](https://mlsanity.com/paper/2204.00771/) (2022)
- [FedTherapist: Mental Health Monitoring with User-Generated Linguistic Expressions on Smartphones via Federated Learning](https://mlsanity.com/paper/2310.16538/) (2023)
- [Personalized Keyphrase Detection using Speaker and Environment Information](https://mlsanity.com/paper/2104.13970/) (2021)
- [Online Neural Diarization of Unlimited Numbers of Speakers Using Global and Local Attractors](https://mlsanity.com/paper/2206.02432/) (2022)
- [Enabling Resource-efficient AIoT System with Cross-level Optimization: A survey](https://mlsanity.com/paper/2309.15467/) (2023)
- [Configurable Privacy-Preserving Automatic Speech Recognition](https://mlsanity.com/paper/2104.00766/) (2021)
- [A Neural State-Space Model Approach to Efficient Speech Separation](https://mlsanity.com/paper/2305.16932/) (2023)
- [Multi-user VoiceFilter-Lite via Attentive Speaker Embedding](https://mlsanity.com/paper/2107.01201/) (2021)
- [End-to-End Joint Target and Non-Target Speakers ASR](https://mlsanity.com/paper/2306.02273/) (2023)
- [A Composite Predictive-Generative Approach to Monaural Universal Speech Enhancement](https://mlsanity.com/paper/2505.24576/) (2025)
- [Knowledge Distillation for Neural Transducer-based Target-Speaker ASR: Exploiting Parallel Mixture/Single-Talker Speech Data](https://mlsanity.com/paper/2305.15971/) (2023)
- [BeamTransformer: Microphone Array-based Overlapping Speech Detection](https://mlsanity.com/paper/2109.04049/) (2021)
- [SVVAD: Personal Voice Activity Detection for Speaker Verification](https://mlsanity.com/paper/2305.19581/) (2023)
- [Convoifilter: A case study of doing cocktail party speech recognition](https://mlsanity.com/paper/2308.11380/) (2023)
- [Plug-and-Play Co-Occurring Face Attention for Robust Audio-Visual Speaker Extraction](https://mlsanity.com/paper/2505.20635/) (2025)
- [Closing the Gap between Single-User and Multi-User VoiceFilter-Lite](https://mlsanity.com/paper/2202.12169/) (2022)
- [Analysis of impact of emotions on target speech extraction and speech separation](https://mlsanity.com/paper/2208.07091/) (2022)
- [Personalized Speech Enhancement Without a Separate Speaker Embedding Model](https://mlsanity.com/paper/2406.09928/) (2024)
- [Speech Separation with Pretrained Frontend to Minimize Domain Mismatch](https://mlsanity.com/paper/2411.03085/) (2024)
- [HiFi-Stream: Streaming Speech Enhancement with Generative Adversarial Networks](https://mlsanity.com/paper/2503.17141/) (2025)
- [StarTSE: Towards Streaming Target Speaker Extraction via Chunk-wise Interleaved Splicing of Autoregressive Language Model](https://mlsanity.com/paper/2604.19635/) (2026)
- [Breaking the trade-off in personalized speech enhancement with cross-task knowledge distillation](https://mlsanity.com/paper/2211.02944/) (2022)
- [Improving Target Speaker Extraction with Sparse LDA-transformed Speaker Embeddings](https://mlsanity.com/paper/2301.06277/) (2023)
- [Generative Model for Models: Rapid DNN Customization for Diverse Tasks and Resource Constraints](https://mlsanity.com/paper/2308.15003/) (2023)
- [Causal Self-supervised Pretrained Frontend with Predictive Code for Speech Separation](https://mlsanity.com/paper/2504.02302/) (2025)

## Abstract (arXiv)

We introduce VoiceFilter-Lite, a single-channel source separation model that runs on the device to preserve only the speech signals from a target user, as part of a streaming speech recognition system. Delivering such a model presents numerous challenges: It should improve the performance when the input signal consists of overlapped speech, and must not hurt the speech recognition performance under all other acoustic conditions. Besides, this model must be tiny, fast, and perform inference in a streaming fashion, in order to have minimal impact on CPU, memory, battery and latency. We propose novel techniques to meet these multi-faceted requirements, including using a new asymmetric loss, and adopting adaptive runtime suppression strength. We also show that such a model can be quantized as a 8-bit integer model and run in realtime.