# Multimodal generative semantic communication based on latent diffusion model

arXiv: https://arxiv.org/abs/2408.05455 · HTML: https://mlsanity.com/paper/2408.05455/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion for compression and wireless, one of 14 directions in Diffusion models.
- Builds on 10 of its 15 references that are in the map.
- First posted on arXiv 10 Aug 2024.

## Placed in

- Diffusion models / [Diffusion for compression and wireless](https://mlsanity.com/diffusion/diffusion-for-compression-and-wireless/)

## Builds on (most cited first, 10 of 10)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Deep Learning Enabled Semantic Communication Systems](https://mlsanity.com/paper/2006.10685/) (2020)
- [Semantic Communication Systems for Speech Transmission](https://mlsanity.com/paper/2102.12605/) (2021)
- [Task-Oriented Multi-User Semantic Communications](https://mlsanity.com/paper/2112.10255/) (2021)
- [Generative Joint Source-Channel Coding for Semantic Image Transmission](https://mlsanity.com/paper/2211.13772/) (2022)
- [Task-Oriented Multi-User Semantic Communications for VQA Task](https://mlsanity.com/paper/2108.07357/) (2021)
- [A Unified Multi-Task Semantic Communication System for Multimodal Data](https://mlsanity.com/paper/2209.07689/) (2022)
- [Generative Semantic Communication: Diffusion Models Beyond Bit Recovery](https://mlsanity.com/paper/2306.04321/) (2023)

## Built on by (most cited first, 3 of 3)

- [Generative Diffusion Models for Wireless Networks: Fundamental, Architecture, and State-of-the-Art](https://mlsanity.com/paper/2507.16733/) (2025)
- [Diffusion Models for Future Networks and Communications: A Comprehensive Survey](https://mlsanity.com/paper/2508.01586/) (2025)
- [LaMI-GO: Latent Mixture Integration for Goal-Oriented Communications Achieving High Spectrum Efficiency](https://mlsanity.com/paper/2412.17839/) (2024)

## Abstract (arXiv)

In emergencies, the ability to quickly and accurately gather environmental data and command information, and to make timely decisions, is particularly critical. Traditional semantic communication frameworks, primarily based on a single modality, are susceptible to complex environments and lighting conditions, thereby limiting decision accuracy. To this end, this paper introduces a multimodal generative semantic communication framework named mm-GESCO. The framework ingests streams of visible and infrared modal image data, generates fused semantic segmentation maps, and transmits them using a combination of one-hot encoding and zlib compression techniques to enhance data transmission efficiency. At the receiving end, the framework can reconstruct the original multimodal images based on the semantic maps. Additionally, a latent diffusion model based on contrastive learning is designed to align different modal data within the latent space, allowing mm-GESCO to reconstruct latent features of any modality presented at the input. Experimental results demonstrate that mm-GESCO achieves a compression ratio of up to 200 times, surpassing the performance of existing semantic communication frameworks and exhibiting excellent performance in downstream tasks such as object classification and detection.