# Multimodal Speech Enhancement Using Burst Propagation

arXiv: https://arxiv.org/abs/2209.03275 · HTML: https://mlsanity.com/paper/2209.03275/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 26 references that are in the map.
- First posted on arXiv 7 Sep 2022.

## Builds on (most cited first, 5 of 5)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Multimodal Audio-Visual Information Fusion using Canonical-Correlated Graph Neural Network for Energy-Efficient Speech Enhancement](https://mlsanity.com/paper/2202.04528/) (2022)
- [Context-sensitive neocortical neurons transform the effectiveness and efficiency of neural information processing](https://mlsanity.com/paper/2207.07338/) (2022)
- [Real-Time Lightweight Chaotic Encryption for 5G IoT Enabled Lip-Reading Driven Secure Hearing-Aid](https://mlsanity.com/paper/1809.04966/) (2018)
- [On the Importance of Super-Gaussian Speech Priors for Machine-Learning Based Speech Enhancement](https://mlsanity.com/paper/1703.05003/) (2017)

## Abstract (arXiv)

This paper proposes the MBURST, a novel multimodal solution for audio-visual speech enhancements that consider the most recent neurological discoveries regarding pyramidal cells of the prefrontal cortex and other brain regions. The so-called burst propagation implements several criteria to address the credit assignment problem in a more biologically plausible manner: steering the sign and magnitude of plasticity through feedback, multiplexing the feedback and feedforward information across layers through different weight connections, approximating feedback and feedforward connections, and linearizing the feedback signals. MBURST benefits from such capabilities to learn correlations between the noisy signal and the visual stimuli, thus attributing meaning to the speech by amplifying relevant information and suppressing noise. Experiments conducted over a Grid Corpus and CHiME3-based dataset show that MBURST can reproduce similar mask reconstructions to the multimodal backpropagation-based baseline while demonstrating outstanding energy efficiency management, reducing the neuron firing rates to values up to 70% lower. Such a feature implies more sustainable implementations, suitable and desirable for hearing aids or any other similar embedded systems.