# Time-Domain Multi-modal Bone/air Conducted Speech Enhancement

arXiv: https://arxiv.org/abs/1911.09847 · HTML: https://mlsanity.com/paper/1911.09847/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Nov 2019.

## Built on by (most cited first, 6 of 6)

- [EMA2S: An End-to-End Multimodal Articulatory-to-Speech System](https://mlsanity.com/paper/2102.03786/) (2021)
- [EPG2S: Speech Generation and Speech Enhancement based on Electropalatography and Audio Signals using Multimodal Learning](https://mlsanity.com/paper/2206.07860/) (2022)
- [EMGSE: Acoustic/EMG Fusion for Multimodal Speech Enhancement](https://mlsanity.com/paper/2202.06507/) (2022)
- [Modality-Specific Speech Enhancement and Noise-Adaptive Fusion for Acoustic and Body-Conduction Microphone Framework](https://mlsanity.com/paper/2508.17336/) (2025)
- [Bone-conduction Guided Multimodal Speech Enhancement with Conditional Diffusion Models](https://mlsanity.com/paper/2601.12354/) (2026)
- [NasoVoce: A Nose-Mounted Low-Audibility Speech Interface for Always-Available Speech Interaction](https://mlsanity.com/paper/2603.10324/) (2026)

## Abstract (arXiv)

Previous studies have proven that integrating video signals, as a complementary modality, can facilitate improved performance for speech enhancement (SE). However, video clips usually contain large amounts of data and pose a high cost in terms of computational resources and thus may complicate the SE system. As an alternative source, a bone-conducted speech signal has a moderate data size while manifesting speech-phoneme structures, and thus complements its air-conducted counterpart. In this study, we propose a novel multi-modal SE structure in the time domain that leverages bone- and air-conducted signals. In addition, we examine two ensemble-learning-based strategies, early fusion (EF) and late fusion (LF), to integrate the two types of speech signals, and adopt a deep learning-based fully convolutional network to conduct the enhancement. The experiment results on the Mandarin corpus indicate that this newly presented multi-modal (integrating bone- and air-conducted signals) SE structure significantly outperforms the single-source SE counterparts (with a bone- or air-conducted signal only) in various speech evaluation metrics. In addition, the adoption of an LF strategy other than an EF in this novel SE multi-modal structure achieves better results.