# Towards Dynamic Neural Communication and Speech Neuroprosthesis Based on Viseme Decoding

arXiv: https://arxiv.org/abs/2501.14790 · HTML: https://mlsanity.com/paper/2501.14790/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 39 references that are in the map.
- First posted on arXiv 9 Jan 2025.

## Builds on (most cited first, 12 of 12)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [KAN: Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2404.19756/) (2024)
- [EEGNet: A Compact Convolutional Network for EEG-based Brain-Computer Interfaces](https://mlsanity.com/paper/1611.08024/) (2016)
- [Audio-Driven Emotional Video Portraits](https://mlsanity.com/paper/2104.07452/) (2021)
- [Realistic Speech-Driven Facial Animation with GANs](https://mlsanity.com/paper/1906.06337/) (2019)
- [FBCNet: A Multi-view Convolutional Neural Network for Brain-Computer Interface](https://mlsanity.com/paper/2104.01233/) (2021)
- [Unsupervised Representation Learning from Pre-trained Diffusion Probabilistic Models](https://mlsanity.com/paper/2212.12990/) (2022)
- [Towards Voice Reconstruction from EEG during Imagined Speech](https://mlsanity.com/paper/2301.07173/) (2023)
- [Digital Voicing of Silent Speech](https://mlsanity.com/paper/2010.02960/) (2020)
- [Diff-E: Diffusion-based Learning for Decoding Imagined Speech EEG](https://mlsanity.com/paper/2307.14389/) (2023)
- [Toward Imagined Speech based Smart Communication System: Potential Applications on Metaverse Conditions](https://mlsanity.com/paper/2112.08569/) (2021)

## Built on by (most cited first, 2 of 2)

- [Foundation Models for Cross-Domain EEG Analysis Application: A Survey](https://mlsanity.com/paper/2508.15716/) (2025)
- [Toward Practical BCI: A Real-time Wireless Imagined Speech EEG Decoding System](https://mlsanity.com/paper/2511.07936/) (2025)

## Abstract (arXiv)

Decoding text, speech, or images from human neural signals holds promising potential both as neuroprosthesis for patients and as innovative communication tools for general users. Although neural signals contain various information on speech intentions, movements, and phonetic details, generating informative outputs from them remains challenging, with mostly focusing on decoding short intentions or producing fragmented outputs. In this study, we developed a diffusion model-based framework to decode visual speech intentions from speech-related non-invasive brain signals, to facilitate face-to-face neural communication. We designed an experiment to consolidate various phonemes to train visemes of each phoneme, aiming to learn the representation of corresponding lip formations from neural signals. By decoding visemes from both isolated trials and continuous sentences, we successfully reconstructed coherent lip movements, effectively bridging the gap between brain signals and dynamic visual interfaces. The results highlight the potential of viseme decoding and talking face reconstruction from human neural signals, marking a significant step toward dynamic neural communication systems and speech neuroprosthesis for patients.