# End-to-end audio-visual learning for cochlear implant sound coding simulations in noisy environments

arXiv: https://arxiv.org/abs/2508.13576 · HTML: https://mlsanity.com/paper/2508.13576/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 36 references that are in the map.
- First posted on arXiv 19 Aug 2025.

## Builds on (most cited first, 13 of 13)

- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation](https://mlsanity.com/paper/2008.09586/) (2020)
- [Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks](https://mlsanity.com/paper/1709.00944/) (2017)
- [Audio-visual Speech Enhancement Using Conditional Variational Auto-Encoders](https://mlsanity.com/paper/1908.02590/) (2019)
- [IndicVoices-R: Unlocking a Massive Multilingual Multi-speaker Speech Corpus for Scaling Indian TTS](https://mlsanity.com/paper/2409.05356/) (2024)
- [Lip-Reading Driven Deep Learning Approach for Speech Enhancement](https://mlsanity.com/paper/1808.00046/) (2018)
- [Lite Audio-Visual Speech Enhancement](https://mlsanity.com/paper/2005.11769/) (2020)
- [Improved Lite Audio-Visual Speech Enhancement](https://mlsanity.com/paper/2008.13222/) (2020)
- [Universal Speech Enhancement with Regression and Generative Mamba](https://mlsanity.com/paper/2505.21198/) (2025)
- [Leveraging Self-Supervised Audio-Visual Pretrained Models to Improve Vocoded Speech Intelligibility in Cochlear Implant Simulation](https://mlsanity.com/paper/2307.07748/) (2023)
- [Bridging The Multi-Modality Gaps of Audio, Visual and Linguistic for Speech Enhancement](https://mlsanity.com/paper/2501.13375/) (2025)
- [ElectrodeNet -- A Deep Learning Based Sound Coding Strategy for Cochlear Implants](https://mlsanity.com/paper/2305.16753/) (2023)
- [NeuroAMP: A Novel End-to-end General Purpose Deep Neural Amplifier for Personalized Hearing Aids](https://mlsanity.com/paper/2502.10822/) (2025)

## Abstract (arXiv)

The cochlear implant (CI) is a successful biomedical device that enables individuals with severe-to-profound hearing loss to perceive sound through electrical stimulation, yet listening in noise remains challenging. Recent deep learning advances offer promising potential for CI sound coding by integrating visual cues. In this study, an audio-visual speech enhancement (AVSE) module is integrated with the ElectrodeNet-CS (ECS) model to form the end-to-end CI system, AVSE-ECS. Simulations show that the AVSE-ECS system with joint training achieves high objective speech intelligibility and improves the signal-to-error ratio (SER) by 7.4666 dB compared to the advanced combination encoder (ACE) strategy. These findings underscore the potential of AVSE-based CI sound coding.