# Multimodal Semantic Communication for Generative Audio-Driven Video Conferencing

arXiv: https://arxiv.org/abs/2410.22112 · HTML: https://mlsanity.com/paper/2410.22112/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2024.

## Built on by (most cited first, 5 of 5)

- [Conquering High Packet-Loss Erasure: MoE Swin Transformer-Based Video Semantic Communication](https://mlsanity.com/paper/2508.01205/) (2025)
- [Unmasking Puppeteers: Leveraging Biometric Leakage to Expose Impersonation in AI-Based Videoconferencing](https://mlsanity.com/paper/2510.03548/) (2025)
- [STSA: Spatial-Temporal Semantic Alignment for Visual Dubbing](https://mlsanity.com/paper/2503.23039/) (2025)
- [Large AI Model-Enabled Generative Semantic Communications for Image Transmission](https://mlsanity.com/paper/2509.21394/) (2025)
- [Semantic Compression via Multimodal Representation Learning](https://mlsanity.com/paper/2509.24431/) (2025)

## Abstract (arXiv)

This paper studies an efficient multimodal data communication scheme for video conferencing. In our considered system, a speaker gives a talk to the audiences, with talking head video and audio being transmitted. Since the speaker does not frequently change posture and high-fidelity transmission of audio (speech and music) is required, redundant visual video data exists and can be removed by generating the video from the audio. To this end, we propose a wave-to-video (Wav2Vid) system, an efficient video transmission framework that reduces transmitted data by generating talking head video from audio. In particular, full-duration audio and short-duration video data are synchronously transmitted through a wireless channel, with neural networks (NNs) extracting and encoding audio and video semantics. The receiver then combines the decoded audio and video data, as well as uses a generative adversarial network (GAN) based model to generate the lip movement videos of the speaker. Simulation results show that the proposed Wav2Vid system can reduce the amount of transmitted data by up to 83% while maintaining the perceptual quality of the generated conferencing video.