# Dynamic Neural Communication: Convergence of Computer Vision and Brain-Computer Interface

arXiv: https://arxiv.org/abs/2411.09211 · HTML: https://mlsanity.com/paper/2411.09211/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 32 references that are in the map.
- First posted on arXiv 14 Nov 2024.

## Builds on (most cited first, 10 of 10)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [KAN: Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2404.19756/) (2024)
- [Audio-Driven Emotional Video Portraits](https://mlsanity.com/paper/2104.07452/) (2021)
- [VisemeNet: Audio-Driven Animator-Centric Speech Animation](https://mlsanity.com/paper/1805.09488/) (2018)
- [DreamDiffusion: Generating High-Quality Images from Brain EEG Signals](https://mlsanity.com/paper/2306.16934/) (2023)
- [Unsupervised Representation Learning from Pre-trained Diffusion Probabilistic Models](https://mlsanity.com/paper/2212.12990/) (2022)
- [Towards Voice Reconstruction from EEG during Imagined Speech](https://mlsanity.com/paper/2301.07173/) (2023)
- [Diff-E: Diffusion-based Learning for Decoding Imagined Speech EEG](https://mlsanity.com/paper/2307.14389/) (2023)
- [Toward Imagined Speech based Smart Communication System: Potential Applications on Metaverse Conditions](https://mlsanity.com/paper/2112.08569/) (2021)

## Abstract (arXiv)

Interpreting human neural signals to decode static speech intentions such as text or images and dynamic speech intentions such as audio or video is showing great potential as an innovative communication tool. Human communication accompanies various features, such as articulatory movements, facial expressions, and internal speech, all of which are reflected in neural signals. However, most studies only generate short or fragmented outputs, while providing informative communication by leveraging various features from neural signals remains challenging. In this study, we introduce a dynamic neural communication method that leverages current computer vision and brain-computer interface technologies. Our approach captures the user's intentions from neural signals and decodes visemes in short time steps to produce dynamic visual outputs. The results demonstrate the potential to rapidly capture and reconstruct lip movements during natural speech attempts from human neural signals, enabling dynamic neural communication through the convergence of computer vision and brain--computer interface.