# Stream-Voice-Anon: Enhancing Utility of Real-Time Speaker Anonymization via Neural Audio Codec and Language Models

arXiv: https://arxiv.org/abs/2601.13948 · HTML: https://mlsanity.com/paper/2601.13948/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 35 references that are in the map.
- First posted on arXiv 20 Jan 2026.

## Builds on (most cited first, 29 of 29)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [Seen and Unseen emotional style transfer for voice conversion with a new emotional speech dataset](https://mlsanity.com/paper/2010.14794/) (2020)
- [Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens](https://mlsanity.com/paper/2503.01710/) (2025)
- [CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Masking](https://mlsanity.com/paper/2303.00332/) (2023)
- [Libriheavy: a 50,000 hours ASR corpus with punctuation casing and context](https://mlsanity.com/paper/2309.08105/) (2023)
- [Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis](https://mlsanity.com/paper/2411.01156/) (2024)
- [The VoicePrivacy 2024 Challenge Evaluation Plan](https://mlsanity.com/paper/2404.02677/) (2024)
- [Speaker anonymisation using the McAdams coefficient](https://mlsanity.com/paper/2011.01130/) (2020)
- [Anonymizing Speech: Evaluating and Designing Speaker Anonymization Techniques](https://mlsanity.com/paper/2308.04455/) (2023)
- [Speaker anonymization using neural audio codec language models](https://mlsanity.com/paper/2309.14129/) (2023)
- [Anonymizing Speech with Generative Adversarial Networks to Preserve Speaker Privacy](https://mlsanity.com/paper/2210.07002/) (2022)
- [Speaker anonymization using orthogonal Householder neural network](https://mlsanity.com/paper/2305.18823/) (2023)
- [NPU-NTU System for Voice Privacy 2024 Challenge](https://mlsanity.com/paper/2409.04173/) (2024)
- [Improving speaker de-identification with functional data analysis of f0 trajectories](https://mlsanity.com/paper/2203.16738/) (2022)
- [StreamVoice: Streamable Context-Aware Language Modeling for Real-time Zero-Shot Voice Conversion](https://mlsanity.com/paper/2401.11053/) (2024)
- [End-to-end streaming model for low-latency speech anonymization](https://mlsanity.com/paper/2406.09277/) (2024)
- [DarkStream: real-time speech anonymization with low latency](https://mlsanity.com/paper/2509.04667/) (2025)
- [NTU-NPU System for Voice Privacy 2024 Challenge](https://mlsanity.com/paper/2410.02371/) (2024)
- [EASY: Emotion-aware Speaker Anonymization via Factorized Distillation](https://mlsanity.com/paper/2505.15004/) (2025)
- [Use Cases for Voice Anonymization](https://mlsanity.com/paper/2508.06356/) (2025)
- [StreamVoice+: Evolving into End-to-end Streaming Zero-shot Voice Conversion](https://mlsanity.com/paper/2408.02178/) (2024)
- [Preserving spoken content in voice anonymisation with character-level vocoder conditioning](https://mlsanity.com/paper/2408.04306/) (2024)

## Built on by (most cited first, 7 of 7)

- [StreamVoiceAnon+: Emotion-Preserving Streaming Speaker Anonymization via Frame-Level Acoustic Distillation](https://mlsanity.com/paper/2603.06079/) (2026)
- [Privacy-Preserving End-to-End Full-Duplex Speech Dialogue Models](https://mlsanity.com/paper/2603.08179/) (2026)
- [PHONOS: PHOnetic Neutralization for Online Streaming Applications](https://mlsanity.com/paper/2603.27001/) (2026)
- [NouveauVoice: Generating Novel Pseudo Speakers for Voice Anonymization](https://mlsanity.com/paper/2607.03985/) (2026)
- [Faynt: Scaling and Optimizing Policies for Competitive Melee](https://mlsanity.com/paper/2610.02144/) (2026)
- [LiveAnon: Improving Privacy and Utility in Streaming Speaker Anonymization with Codec Language Models](https://mlsanity.com/paper/2610.15379/) (2026)
- [From Acoustic Sensing to Physical Actions: A Comprehensive Survey of Security Risks in Edge-Deployed Audio LLM-Based Agents](https://mlsanity.com/paper/2610.15390/) (2026)

## Abstract (arXiv)

Protecting speaker identity is crucial for online voice applications, yet streaming speaker anonymization (SA) remains underexplored. Recent research has demonstrated that neural audio codec (NAC) provides superior speaker feature disentanglement and linguistic fidelity. NAC can also be used with causal language models (LM) to enhance linguistic fidelity and prompt control for streaming tasks. However, existing NAC-based online LM systems are designed for voice conversion (VC) rather than anonymization, lacking the techniques required for privacy protection. Building on these advances, we present Stream-Voice-Anon, which adapts modern causal LM-based NAC architectures specifically for streaming SA by integrating anonymization techniques. Our anonymization approach incorporates pseudo-speaker representation sampling, a speaker embedding mixing and diverse prompt selection strategies for LM conditioning that leverage the disentanglement properties of quantized content codes to prevent speaker information leakage. Additionally, we compare dynamic and fixed delay configurations to explore latency-privacy trade-offs in real-time scenarios. Under the VoicePrivacy 2024 Challenge protocol, Stream-Voice-Anon achieves substantial improvements in intelligibility (up to 46% relative WER reduction) and emotion preservation (up to 28% UAR relative) compared to the previous state-of-the-art streaming method DarkStream while maintaining comparable latency (180ms vs 200ms) and privacy protection against lazy-informed attackers, though showing 15% relative degradation against semi-informed attackers.