# CochleaNet: A Robust Language-independent Audio-Visual Model for Speech Enhancement

arXiv: https://arxiv.org/abs/1909.10407 · HTML: https://mlsanity.com/paper/1909.10407/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Sep 2019.

## Built on by (most cited first, 5 of 5)

- [Multimodal Audio-Visual Information Fusion using Canonical-Correlated Graph Neural Network for Energy-Efficient Speech Enhancement](https://mlsanity.com/paper/2202.04528/) (2022)
- [Context-sensitive neocortical neurons transform the effectiveness and efficiency of neural information processing](https://mlsanity.com/paper/2207.07338/) (2022)
- [A Novel Context-Aware Multimodal Framework for Persian Sentiment Analysis](https://mlsanity.com/paper/2103.02636/) (2021)
- [Audio Does Matter: Importance-Aware Multi-Granularity Fusion for Video Moment Retrieval](https://mlsanity.com/paper/2508.04273/) (2025)
- [Canonical Cortical Graph Neural Networks and its Application for Speech Enhancement in Audio-Visual Hearing Aids](https://mlsanity.com/paper/2206.02671/) (2022)

## Abstract (arXiv)

Noisy situations cause huge problems for suffers of hearing loss as hearing aids often make the signal more audible but do not always restore the intelligibility. In noisy settings, humans routinely exploit the audio-visual (AV) nature of the speech to selectively suppress the background noise and to focus on the target speaker. In this paper, we present a causal, language, noise and speaker independent AV deep neural network (DNN) architecture for speech enhancement (SE). The model exploits the noisy acoustic cues and noise robust visual cues to focus on the desired speaker and improve the speech intelligibility. To evaluate the proposed SE framework a first of its kind AV binaural speech corpus, called ASPIRE, is recorded in real noisy environments including cafeteria and restaurant. We demonstrate superior performance of our approach in terms of objective measures and subjective listening tests over the state-of-the-art SE approaches as well as recent DNN based SE models. In addition, our work challenges a popular belief that a scarcity of multi-language large vocabulary AV corpus and wide variety of noises is a major bottleneck to build a robust language, speaker and noise independent SE systems. We show that a model trained on synthetic mixture of Grid corpus (with 33 speakers and a small English vocabulary) and ChiME 3 Noises (consisting of only bus, pedestrian, cafeteria, and street noises) generalise well not only on large vocabulary corpora but also on completely unrelated languages (such as Mandarin), wide variety of speakers and noises.