# Canonical Cortical Graph Neural Networks and its Application for Speech Enhancement in Audio-Visual Hearing Aids

arXiv: https://arxiv.org/abs/2206.02671 · HTML: https://mlsanity.com/paper/2206.02671/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 41 references that are in the map.
- First posted on arXiv 6 Jun 2022.

## Builds on (most cited first, 9 of 9)

- [Deep Graph Contrastive Representation Learning](https://mlsanity.com/paper/2006.04131/) (2020)
- [Understanding self-supervised Learning Dynamics without Contrastive Pairs](https://mlsanity.com/paper/2102.06810/) (2021)
- [From Canonical Correlation Analysis to Self-supervised Graph Neural Networks](https://mlsanity.com/paper/2106.12484/) (2021)
- [Lip-Reading Driven Deep Learning Approach for Speech Enhancement](https://mlsanity.com/paper/1808.00046/) (2018)
- [Multimodal Audio-Visual Information Fusion using Canonical-Correlated Graph Neural Network for Energy-Efficient Speech Enhancement](https://mlsanity.com/paper/2202.04528/) (2022)
- [CochleaNet: A Robust Language-independent Audio-Visual Model for Speech Enhancement](https://mlsanity.com/paper/1909.10407/) (2019)
- [Contextual Audio-Visual Switching For Speech Enhancement in Real-World Environments](https://mlsanity.com/paper/1808.09825/) (2018)
- [Real-Time Lightweight Chaotic Encryption for 5G IoT Enabled Lip-Reading Driven Secure Hearing-Aid](https://mlsanity.com/paper/1809.04966/) (2018)
- [A Novel Speech Intelligibility Enhancement Model based on CanonicalCorrelation and Deep Learning](https://mlsanity.com/paper/2202.05756/) (2022)

## Built on by (most cited first, 1 of 1)

- [BioNeRF: Biologically Plausible Neural Radiance Fields for View Synthesis](https://mlsanity.com/paper/2402.07310/) (2024)

## Abstract (arXiv)

Despite the recent success of machine learning algorithms, most models face drawbacks when considering more complex tasks requiring interaction between different sources, such as multimodal input data and logical time sequences. On the other hand, the biological brain is highly sharpened in this sense, empowered to automatically manage and integrate such streams of information. In this context, this work draws inspiration from recent discoveries in brain cortical circuits to propose a more biologically plausible self-supervised machine learning approach. This combines multimodal information using intra-layer modulations together with Canonical Correlation Analysis, and a memory mechanism to keep track of temporal data, the overall approach termed Canonical Cortical Graph Neural networks. This is shown to outperform recent state-of-the-art models in terms of clean audio reconstruction and energy efficiency for a benchmark audio-visual speech dataset. The enhanced performance is demonstrated through a reduced and smother neuron firing rate distribution. suggesting that the proposed model is amenable for speech enhancement in future audio-visual hearing aid devices.