mlsanityData to 14 Oct 2026

Disentangling speech from surroundings with neural embeddings

arXiv 2203.15578 ↗First posted 29 Mar 2022cs.SD cs.LG eess.AScited by 6

Not on the map. Why?

Abstract (arXiv)

We present a method to separate speech signals from noisy environments in the embedding space of a neural audio codec. We introduce a new training procedure that allows our model to produce structured encodings of audio waveforms given by embedding vectors, where one part of the embedding vector represents the speech signal, and the rest represent the environment. We achieve this by partitioning the embeddings of different input waveforms and training the model to faithfully reconstruct audio from mixed partitions, thereby ensuring each partition encodes a separate audio attribute. As use cases, we demonstrate the separation of speech from background noise or from reverberation characteristics. Our method also allows for targeted adjustments of the audio output characteristics.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 23 of its 32 references that are in the map. First posted on arXiv 29 Mar 2022.

Builds on

23 of 32 references.

References · 23

  1. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  2. 2018Towards a Definition of Disentangled Representationscited by 296
  3. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  4. 2020Swapping Autoencoder for Deep Image Manipulationcited by 215
  5. 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 192
  6. 2020DDSP: Differentiable Digital Signal Processingcited by 178
  7. 2015Deep Convolutional Inverse Graphics Networkcited by 121
  8. 2017Fader Networks: Manipulating Images by Sliding Attributescited by 118
  9. 2020Unsupervised Speech Decomposition via Triple Information Bottleneckcited by 88
  10. 2020ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metriccited by 79
  11. 2020A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluationcited by 52
  12. 2019Low Bit-Rate Speech Coding with VQ-VAE and a WaveNet Decodercited by 49
11 more
  1. 2020A Spectral Energy Distance for Parallel Speech Synthesiscited by 42
  2. 2019Universal Sound Separationcited by 41
  3. 2019SPICE: Self-supervised Pitch Estimationcited by 27
  4. 2018Multi-target Voice Conversion without Parallel Data by Adversarially Learning Disentangled Audio Representationscited by 25
  5. 2017End-to-End Optimized Speech Coding with Deep Neural Networkscited by 22
  6. 2018A Universal Music Translation Networkcited by 22
  7. 2017Disentangling Factors of Variation by Mixing Themcited by 18
  8. 2021NVC-Net: End-to-End Adversarial Voice Conversioncited by 9
  9. 2020Source-Aware Neural Speech Coding for Noisy Speech Compressioncited by 5
  10. 2020DisCont: Self-Supervised Visual Attribute Disentanglement using Context Vectorscited by 2
  11. 2018Towards Learning Fine-Grained Disentangled Representations from Speechcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 1; 2025: 2; 2026: 1.

6 papers.

  1. 2023Generative De-Quantization for Neural Speech Codec via Latent Diffusioncited by 9
  2. 2024Learning Source Disentanglement in Neural Audio Codeccited by 6
  3. 2023AudioFormer: Audio Transformer learns audio feature representations from discrete acoustic codescited by 1
  4. 2025Learning Linearity in Audio Consistency Autoencoders via Implicit Regularizationcited by 1
  5. 2025Explainable Disentanglement on Discrete Speech Representations for Noise-Robust ASRcited by 1
  6. 2026PoDAR: Power-Disentangled Audio Representation for Generative Modelingcited by 0