mlsanityData to 7 Oct 2026

SEANet: A Multi-modal Speech Enhancement Network

arXiv 2009.02095 ↗First posted 4 Sep 2020eess.AS cs.LG cs.SDcited by 35

Not on the map. Why?

Abstract (arXiv)

We explore the possibility of leveraging accelerometer data to perform speech enhancement in very noisy conditions. Although it is possible to only partially reconstruct user's speech from the accelerometer, the latter provides a strong conditioning signal that is not influenced from noise sources in the environment. Based on this observation, we feed a multi-modal input to SEANet (Sound EnhAncement Network), a wave-to-wave fully convolutional model, which adopts a combination of feature losses and adversarial losses to reconstruct an enhanced version of user's speech. We trained our model with data collected by sensors mounted on an earbud and synthetically corrupted by adding different kinds of noise sources to the audio signal. Our experimental results demonstrate that it is possible to achieve very high quality results, even in the case of interfering speech at the same level of loudness. A sample of the output produced by our model is available at https://google-research.github.io/seanet/multimodal/speech.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 7; 2023: 3; 2024: 8; 2025: 11; 2026: 5.

35 papers.

Sort:

By area:

  1. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  2. 2022High Fidelity Neural Audio Compressioncited by 673
  3. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  4. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  5. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 77
  6. 2022HiFi++: a Unified Framework for Bandwidth Extension and Speech Enhancementcited by 49
  7. 2023FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codeccited by 44
  8. 2022Multi-instrument Music Synthesis with Spectrogram Diffusioncited by 42
  9. 2024The Codecfake Dataset and Countermeasures for the Universally Detection of Deepfake Audiocited by 31
  10. 2025BrainOmni: A Brain Foundation Model for Unified EEG and MEG Signalscited by 25
  11. 2024The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learningcited by 21
  12. 2022WaveFit: An Iterative and Non-autoregressive Neural Vocoder based on Fixed-Point Iterationcited by 20
23 more
  1. 2025LibriBrain: Over 50 Hours of Within-Subject MEG to Improve Speech Decoding Methods at Scalecited by 20
  2. 2022SpeechPainter: Text-conditioned Speech Inpaintingcited by 17
  3. 2022FFC-SE: Fast Fourier Convolution for Speech Enhancementcited by 7
  4. 2024NeuGPT: Unified multi-modal Neural GPTcited by 6
  5. 2024A Survey of Deep Learning Audio Generation Methodscited by 5
  6. 2024Vibravox: A Dataset of French Speech Captured with Body-conduction Audio Sensorscited by 5
  7. 2024DM-Codec: Distilling Multimodal Representations for Speech Tokenizationcited by 5
  8. 2025Towards a Japanese Full-duplex Spoken Dialogue Systemcited by 5
  9. 2025SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codeccited by 5
  10. 2023AudioSlots: A slot-centric generative model for audio separationcited by 4
  11. 2025SpectroStream: A Versatile Neural Codec for General Audiocited by 4
  12. 2025MEGConformer: Conformer-Based MEG Decoder for Robust Speech and Phoneme Classificationcited by 4
  13. 2025HH-Codec: High Compression High-fidelity Discrete Neural Codec for Spoken Language Modelingcited by 2
  14. 2024Spiking Music: Audio Compression with Event Based Auto-encoderscited by 1
  15. 2025Leveraging Discriminative Latent Representations for Conditioning GAN-Based Speech Enhancementcited by 1
  16. 2026DSA-Tokenizer: Disentangled Semantic-Acoustic Tokenization via Flow Matching-based Hierarchical Fusioncited by 1
  17. 2026Speech Codec Probing from Semantic and Phonetic Perspectivescited by 1
  18. 2025Neural Spectral Band Generation for Audio Codingcited by 0
  19. 2025SUBARU: A Practical Approach to Power Saving in Hearables Using SUB-Nyquist Audio Resolution Upsamplingcited by 0
  20. 2025MF-Speech: Achieving Fine-Grained and Compositional Control in Speech Generation via Factor Disentanglementcited by 0
  21. 2026The Differentiable Auditory Loop (DAL): An ML Framework for Hyper-Personalized Hearing Aidscited by 0
  22. 2026Next-Token Prediction Learns Generalisable Representations of Sleep Physiologycited by 0
  23. 2026NAC: Neural Action Codec for Vision-Language-Action Modelscited by 0