# Cortex Inspired Learning to Recover Damaged Signal Modality with ReD-SOM Model

arXiv: https://arxiv.org/abs/2307.15095 · HTML: https://mlsanity.com/paper/2307.15095/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 48 references that are in the map.
- First posted on arXiv 27 Jul 2023.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [SOM-VAE: Interpretable Discrete Representation Learning on Time Series](https://mlsanity.com/paper/1806.02199/) (2018)
- [Mixture of Inference Networks for VAE-based Audio-visual Speech Enhancement](https://mlsanity.com/paper/1912.10647/) (2019)
- [Machine Learning in Magnetic Resonance Imaging: Image Reconstruction](https://mlsanity.com/paper/2012.05303/) (2020)
- [Improving Self-Organizing Maps with Unsupervised Feature Extraction](https://mlsanity.com/paper/2009.02174/) (2020)
- [SOM-CPC: Unsupervised Contrastive Learning with Self-Organizing Maps for Structured Representations of High-Rate Time Series](https://mlsanity.com/paper/2205.15875/) (2022)
- [Deep Neural Maps](https://mlsanity.com/paper/1810.07291/) (2018)
- [Deep Categorization with Semi-Supervised Self-Organizing Maps](https://mlsanity.com/paper/2006.13682/) (2020)
- [Brain-inspired self-organization with cellular neuromorphic computing for multimodal unsupervised learning](https://mlsanity.com/paper/2004.05488/) (2020)
- [A unified software/hardware scalable architecture for brain-inspired computing based on self-organizing neural models](https://mlsanity.com/paper/2201.02262/) (2022)

## Abstract (arXiv)

Recent progress in the fields of AI and cognitive sciences opens up new challenges that were previously inaccessible to study. One of such modern tasks is recovering lost data of one modality by using the data from another one. A similar effect (called the McGurk Effect) has been found in the functioning of the human brain. Observing this effect, one modality of information interferes with another, changing its perception. In this paper, we propose a way to simulate such an effect and use it to reconstruct lost data modalities by combining Variational Auto-Encoders, Self-Organizing Maps, and Hebb connections in a unified ReD-SOM (Reentering Deep Self-organizing Map) model. We are inspired by human's capability to use different zones of the brain in different modalities, in case of having a lack of information in one of the modalities. This new approach not only improves the analysis of ambiguous data but also restores the intended signal! The results obtained on the multimodal dataset demonstrate an increase of quality of the signal reconstruction. The effect is remarkable both visually and quantitatively, specifically in presence of a significant degree of signal's distortion.