# LinearVC: Linear transformations of self-supervised features through the lens of voice conversion

arXiv: https://arxiv.org/abs/2506.01510 · HTML: https://mlsanity.com/paper/2506.01510/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 39 references that are in the map.
- First posted on arXiv 2 Jun 2025.

## Builds on (most cited first, 29 of 29)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Comparative layer-wise analysis of self-supervised speech models](https://mlsanity.com/paper/2211.03929/) (2022)
- [SoundStorm: Efficient Parallel Audio Generation](https://mlsanity.com/paper/2305.09636/) (2023)
- [An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learning](https://mlsanity.com/paper/2008.03648/) (2020)
- [FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion](https://mlsanity.com/paper/2210.15418/) (2022)
- [Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion](https://mlsanity.com/paper/2008.12527/) (2020)
- [Voice Conversion With Just Nearest Neighbors](https://mlsanity.com/paper/2305.18975/) (2023)
- [A Comparison of Discrete and Soft Speech Units for Improved Voice Conversion](https://mlsanity.com/paper/2111.02392/) (2021)
- [VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment](https://mlsanity.com/paper/2406.07855/) (2024)
- [FragmentVC: Any-to-Any Voice Conversion by End-to-End Extracting and Fusing Fine-Grained Voice Fragments With Attention](https://mlsanity.com/paper/2010.14150/) (2020)
- [DASB - Discrete Audio and Speech Benchmark](https://mlsanity.com/paper/2406.14294/) (2024)
- [How Should We Extract Discrete Audio Tokens from Self-Supervised Models?](https://mlsanity.com/paper/2406.10735/) (2024)
- [Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspaces](https://mlsanity.com/paper/2305.12464/) (2023)
- [Predictions of Subjective Ratings and Spoofing Assessments of Voice Conversion Challenge 2020 Submissions](https://mlsanity.com/paper/2009.03554/) (2020)
- [The VoicePrivacy 2022 Challenge: Progress and Perspectives in Voice Anonymisation](https://mlsanity.com/paper/2407.11516/) (2024)
- [Self-Supervised Models of Speech Infer Universal Articulatory Kinematics](https://mlsanity.com/paper/2310.10788/) (2023)
- [StreamVC: Real-Time Low-Latency Voice Conversion](https://mlsanity.com/paper/2401.03078/) (2024)
- [Orthogonality and isotropy of speaker and phonetic information in self-supervised speech representations](https://mlsanity.com/paper/2406.09200/) (2024)
- [Phoneme Hallucinator: One-shot Voice Conversion via Set Expansion](https://mlsanity.com/paper/2308.06382/) (2023)
- [QuickVC: Any-to-many Voice Conversion Using Inverse Short-time Fourier Transform for Faster Conversion](https://mlsanity.com/paper/2302.08296/) (2023)
- [Unsupervised Rhythm and Voice Conversion of Dysarthric to Healthy Speech for ASR](https://mlsanity.com/paper/2501.10256/) (2025)
- [Voice Conversion for Stuttered Speech, Instruments, Unseen Languages and Textually Described Voices](https://mlsanity.com/paper/2310.08104/) (2023)
- [Optimal Transport Maps are Good Voice Converters](https://mlsanity.com/paper/2411.02402/) (2024)
- [Eta-WavLM: Efficient Speaker Identity Removal in Self-Supervised Speech Representations Using a Simple Linear Equation](https://mlsanity.com/paper/2505.19273/) (2025)
- [A Pilot Study of Applying Sequence-to-Sequence Voice Conversion to Evaluate the Intelligibility of L2 Speech Using a Native Speaker's Shadowings](https://mlsanity.com/paper/2410.02239/) (2024)

## Built on by (most cited first, 3 of 3)

- [Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspaces](https://mlsanity.com/paper/2603.12642/) (2026)
- [Self-supervised Speech Comparison for L2 Phone, Rhythm, and Intonation Scoring](https://mlsanity.com/paper/2607.13721/) (2026)
- [Interpreting Speaker Characteristics in the Dimensions of Self-Supervised Speech Features](https://mlsanity.com/paper/2603.03096/) (2026)

## Abstract (arXiv)

We introduce LinearVC, a simple voice conversion method that sheds light on the structure of self-supervised representations. First, we show that simple linear transformations of self-supervised features effectively convert voices. Next, we probe the geometry of the feature space by constraining the set of allowed transformations. We find that just rotating the features is sufficient for high-quality voice conversion. This suggests that content information is embedded in a low-dimensional subspace which can be linearly transformed to produce a target voice. To validate this hypothesis, we finally propose a method that explicitly factorizes content and speaker information using singular value decomposition; the resulting linear projection with a rank of just 100 gives competitive conversion results. Our work has implications for both practical voice conversion and a broader understanding of self-supervised speech representations. Samples and code: https://www.kamperh.com/linearvc/.