# Visual Language Model based Cross-modal Semantic Communication Systems arXiv: https://arxiv.org/abs/2407.00020 ยท HTML: https://mlsanity.com/paper/2407.00020/ ## Key facts - Cited by 12 papers in the map (data through 14 Oct 2026). - Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. - Builds on 26 of its 44 references that are in the map. - First posted on arXiv 6 May 2024. ## Placed in - Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/) ## Builds on (most cited first, 26 of 26) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021) - [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022) - [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019) - [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021) - [SimVLM: Simple Visual Language Model Pretraining with Weak Supervision](https://mlsanity.com/paper/2108.10904/) (2021) - [Recognize Anything: A Strong Image Tagging Model](https://mlsanity.com/paper/2306.03514/) (2023) - [Deep Learning Enabled Semantic Communication Systems](https://mlsanity.com/paper/2006.10685/) (2020) - [A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT](https://mlsanity.com/paper/2303.04226/) (2023) - [Scaling Up Vision-Language Pre-training for Image Captioning](https://mlsanity.com/paper/2111.12233/) (2021) - [Causal Attention for Vision-Language Tasks](https://mlsanity.com/paper/2103.03493/) (2021) - [DeepJSCC-f: Deep Joint Source-Channel Coding of Images with Feedback](https://mlsanity.com/paper/1911.11174/) (2019) - [Semantic Communications for Future Internet: Fundamentals, Applications, and Challenges](https://mlsanity.com/paper/2207.00427/) (2022) - [CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP](https://mlsanity.com/paper/2110.11316/) (2021) - [Nonlinear Transform Source-Channel Coding for Semantic Communications](https://mlsanity.com/paper/2112.10961/) (2021) - [Large Language Model Enhanced Multi-Agent Systems for 6G Communications](https://mlsanity.com/paper/2312.07850/) (2023) - [A Full Dive into Realizing the Edge-enabled Metaverse: Visions, Enabling Technologies,and Challenges](https://mlsanity.com/paper/2203.05471/) (2022) - [Large AI Model Empowered Multimodal Semantic Communications](https://mlsanity.com/paper/2309.01249/) (2023) - [WITT: A Wireless Image Transmission Transformer for Semantic Communications](https://mlsanity.com/paper/2211.00937/) (2022) - [Deep Learning-Enabled Semantic Communication Systems with Task-Unaware Transmitter and Dynamic Data](https://mlsanity.com/paper/2205.00271/) (2022) - [Towards Semantic Communications: Deep Learning-Based Image Semantic Coding](https://mlsanity.com/paper/2208.04094/) (2022) - [Large AI Model-Based Semantic Communications](https://mlsanity.com/paper/2307.03492/) (2023) - [Continual Variational Autoencoder Learning via Online Cooperative Memorization](https://mlsanity.com/paper/2207.10131/) (2022) - [Distributed Resource Scheduling for Large-Scale MEC Systems: A Multi-Agent Ensemble Deep Reinforcement Learning with Imitation Acceleration](https://mlsanity.com/paper/2005.12364/) (2020) - [Joint Optimization of Deployment and Trajectory in UAV and IRS-Assisted IoT Data Collection System](https://mlsanity.com/paper/2210.15203/) (2022) ## Built on by (most cited first, 12 of 12) - [Generative Semantic Communication: Architectures, Technologies, and Applications](https://mlsanity.com/paper/2412.08642/) (2024) - [Improving Generalizability and Undetectability for Targeted Adversarial Attacks on Multimodal Pre-trained Models](https://mlsanity.com/paper/2509.19994/) (2025) - [ICDM: Interference Cancellation Diffusion Models for Wireless Semantic Communications](https://mlsanity.com/paper/2505.19983/) (2025) - [Large Generative Model-assisted Talking-face Semantic Communication System](https://mlsanity.com/paper/2411.03876/) (2024) - [Editable-DeepSC: Reliable Cross-Modal Semantic Communications for Facial Editing](https://mlsanity.com/paper/2411.15702/) (2024) - [From Large AI Models to Agentic AI: A Tutorial on Future Intelligent Communications](https://mlsanity.com/paper/2505.22311/) (2025) - [Progressive Semantic Communication for Efficient Edge-Cloud Vision-Language Models](https://mlsanity.com/paper/2604.26508/) (2026) - [Exploring Textual Semantics Diversity for Image Transmission in Semantic Communication Systems using Visual Language Model](https://mlsanity.com/paper/2503.19386/) (2025) - [Evaluating Robustness of Vision-Language Models Under Noisy Conditions](https://mlsanity.com/paper/2509.12492/) (2025) - [Large AI Model-Enabled Generative Semantic Communications for Image Transmission](https://mlsanity.com/paper/2509.21394/) (2025) - [A Survey on Semantic Communication for Vision: Categories, Frameworks, Enabling Techniques, and Applications](https://mlsanity.com/paper/2601.22202/) (2026) - [From Semantic to Token Communication: The Next Paradigm for Large-Model-Driven 6G Intelligent Connectivity](https://mlsanity.com/paper/2609.10714/) (2026) ## Abstract (arXiv) Semantic Communication (SC) has emerged as a novel communication paradigm in recent years, successfully transcending the Shannon physical capacity limits through innovative semantic transmission concepts. Nevertheless, extant Image Semantic Communication (ISC) systems face several challenges in dynamic environments, including low semantic density, catastrophic forgetting, and uncertain Signal-to-Noise Ratio (SNR). To address these challenges, we propose a novel Vision-Language Model-based Cross-modal Semantic Communication (VLM-CSC) system. The VLM-CSC comprises three novel components: (1) Cross-modal Knowledge Base (CKB) is used to extract high-density textual semantics from the semantically sparse image at the transmitter and reconstruct the original image based on textual semantics at the receiver. The transmission of high-density semantics contributes to alleviating bandwidth pressure. (2) Memory-assisted Encoder and Decoder (MED) employ a hybrid long/short-term memory mechanism, enabling the semantic encoder and decoder to overcome catastrophic forgetting in dynamic environments when there is a drift in the distribution of semantic features. (3) Noise Attention Module (NAM) employs attention mechanisms to adaptively adjust the semantic coding and the channel coding based on SNR, ensuring the robustness of the CSC system. The experimental simulations validate the effectiveness, adaptability, and robustness of the CSC system.