# SEANet: A Multi-modal Speech Enhancement Network

arXiv: https://arxiv.org/abs/2009.02095 · HTML: https://mlsanity.com/paper/2009.02095/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Sep 2020.

## Built on by (most cited first, 30 of 37)

- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [Recent Advances in Speech Language Models: A Survey](https://mlsanity.com/paper/2410.03751/) (2024)
- [LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT](https://mlsanity.com/paper/2310.04673/) (2023)
- [HiFi++: a Unified Framework for Bandwidth Extension and Speech Enhancement](https://mlsanity.com/paper/2203.13086/) (2022)
- [FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec](https://mlsanity.com/paper/2309.07405/) (2023)
- [Multi-instrument Music Synthesis with Spectrogram Diffusion](https://mlsanity.com/paper/2206.05408/) (2022)
- [The Codecfake Dataset and Countermeasures for the Universally Detection of Deepfake Audio](https://mlsanity.com/paper/2405.04880/) (2024)
- [BrainOmni: A Brain Foundation Model for Unified EEG and MEG Signals](https://mlsanity.com/paper/2505.18185/) (2025)
- [The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning](https://mlsanity.com/paper/2406.04328/) (2024)
- [WaveFit: An Iterative and Non-autoregressive Neural Vocoder based on Fixed-Point Iteration](https://mlsanity.com/paper/2210.01029/) (2022)
- [LibriBrain: Over 50 Hours of Within-Subject MEG to Improve Speech Decoding Methods at Scale](https://mlsanity.com/paper/2506.02098/) (2025)
- [SpeechPainter: Text-conditioned Speech Inpainting](https://mlsanity.com/paper/2202.07273/) (2022)
- [FFC-SE: Fast Fourier Convolution for Speech Enhancement](https://mlsanity.com/paper/2204.03042/) (2022)
- [NeuGPT: Unified multi-modal Neural GPT](https://mlsanity.com/paper/2410.20916/) (2024)
- [AudioSlots: A slot-centric generative model for audio separation](https://mlsanity.com/paper/2305.05591/) (2023)
- [A Survey of Deep Learning Audio Generation Methods](https://mlsanity.com/paper/2406.00146/) (2024)
- [Vibravox: A Dataset of French Speech Captured with Body-conduction Audio Sensors](https://mlsanity.com/paper/2407.11828/) (2024)
- [DM-Codec: Distilling Multimodal Representations for Speech Tokenization](https://mlsanity.com/paper/2410.15017/) (2024)
- [Towards a Japanese Full-duplex Spoken Dialogue System](https://mlsanity.com/paper/2506.02979/) (2025)
- [SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codec](https://mlsanity.com/paper/2508.02849/) (2025)
- [SpectroStream: A Versatile Neural Codec for General Audio](https://mlsanity.com/paper/2508.05207/) (2025)
- [MEGConformer: Conformer-Based MEG Decoder for Robust Speech and Phoneme Classification](https://mlsanity.com/paper/2512.01443/) (2025)
- [WaveVerify: A Novel Audio Watermarking Framework for Media Authentication and Combatting Deepfakes](https://mlsanity.com/paper/2507.21150/) (2025)
- [HH-Codec: High Compression High-fidelity Discrete Neural Codec for Spoken Language Modeling](https://mlsanity.com/paper/2507.18897/) (2025)
- [Spiking Music: Audio Compression with Event Based Auto-encoders](https://mlsanity.com/paper/2402.01571/) (2024)
- [Leveraging Discriminative Latent Representations for Conditioning GAN-Based Speech Enhancement](https://mlsanity.com/paper/2508.20859/) (2025)
- [DSA-Tokenizer: Disentangled Semantic-Acoustic Tokenization via Flow Matching-based Hierarchical Fusion](https://mlsanity.com/paper/2601.09239/) (2026)
- [Speech Codec Probing from Semantic and Phonetic Perspectives](https://mlsanity.com/paper/2603.10371/) (2026)

## Abstract (arXiv)

We explore the possibility of leveraging accelerometer data to perform speech enhancement in very noisy conditions. Although it is possible to only partially reconstruct user's speech from the accelerometer, the latter provides a strong conditioning signal that is not influenced from noise sources in the environment. Based on this observation, we feed a multi-modal input to SEANet (Sound EnhAncement Network), a wave-to-wave fully convolutional model, which adopts a combination of feature losses and adversarial losses to reconstruct an enhanced version of user's speech. We trained our model with data collected by sensors mounted on an earbud and synthetically corrupted by adding different kinds of noise sources to the audio signal. Our experimental results demonstrate that it is possible to achieve very high quality results, even in the case of interfering speech at the same level of loudness. A sample of the output produced by our model is available at https://google-research.github.io/seanet/multimodal/speech.