# Agent-driven Generative Semantic Communication with Cross-Modality and Prediction

arXiv: https://arxiv.org/abs/2404.06997 · HTML: https://mlsanity.com/paper/2404.06997/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 45 references that are in the map.
- First posted on arXiv 10 Apr 2024.

## Builds on (most cited first, 30 of 31)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation](https://mlsanity.com/paper/2309.06380/) (2023)
- [Deep Learning Enabled Semantic Communication Systems](https://mlsanity.com/paper/2006.10685/) (2020)
- [LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation](https://mlsanity.com/paper/2303.17189/) (2023)
- [Semantic Image Synthesis via Diffusion Models](https://mlsanity.com/paper/2207.00050/) (2022)
- [UA-DETRAC: A New Benchmark and Protocol for Multi-Object Detection and Tracking](https://mlsanity.com/paper/1511.04136/) (2015)
- [Semantic Communications: Principles and Challenges](https://mlsanity.com/paper/2201.01389/) (2021)
- [Age of Information: An Introduction and Survey](https://mlsanity.com/paper/2007.08564/) (2020)
- [Wireless Deep Video Semantic Transmission](https://mlsanity.com/paper/2205.13129/) (2022)
- [SwinJSCC: Taming Swin Transformer for Deep Joint Source-Channel Coding](https://mlsanity.com/paper/2308.09361/) (2023)
- [Generative Semantic Communication: Diffusion Models Beyond Bit Recovery](https://mlsanity.com/paper/2306.04321/) (2023)
- [Deep Learning-Enabled Semantic Communication Systems with Task-Unaware Transmitter and Dynamic Data](https://mlsanity.com/paper/2205.00271/) (2022)
- [Semantic Communications in Networked Systems: A Data Significance Perspective](https://mlsanity.com/paper/2103.05391/) (2021)
- [Diffusion models for audio semantic communication](https://mlsanity.com/paper/2309.07195/) (2023)
- [Language-Oriented Communication with Semantic Coding and Knowledge Distillation for Text-to-Image Generation](https://mlsanity.com/paper/2309.11127/) (2023)
- [Curriculum Learning for Goal-Oriented Semantic Communications with a Common Language](https://mlsanity.com/paper/2204.10429/) (2022)
- [A Survey on Semantic Communications for Intelligent Wireless Networks](https://mlsanity.com/paper/2202.03705/) (2022)
- [Age-of-Information vs. Value-of-Information Scheduling for Cellular Networked Control Systems](https://mlsanity.com/paper/1903.05356/) (2019)
- [DeepMA: End-to-end Deep Multiple Access for Wireless Image Transmission in Semantic Communication](https://mlsanity.com/paper/2303.11543/) (2023)
- [Knowledge Base Enabled Semantic Communication: A Generative Perspective](https://mlsanity.com/paper/2311.12443/) (2023)
- [Generative AI-driven Semantic Communication Framework for NextG Wireless Network](https://mlsanity.com/paper/2310.09021/) (2023)
- [Semantic Change Driven Generative Semantic Communication Framework](https://mlsanity.com/paper/2309.12775/) (2023)
- [Task-driven Semantic-aware Green Cooperative Transmission Strategy for Vehicular Networks](https://mlsanity.com/paper/2307.15284/) (2023)
- [Freshness on Demand: Optimizing Age of Information for the Query Process](https://mlsanity.com/paper/2011.00917/) (2020)
- [Age of Actuation in a Wireless Power Transfer System](https://mlsanity.com/paper/2303.00507/) (2023)
- [Scalable AI Generative Content for Vehicular Network Semantic Communication](https://mlsanity.com/paper/2311.13782/) (2023)
- [YOLOv5s-BC: An improved YOLOv5s-based method for real-time apple detection](https://mlsanity.com/paper/2311.05811/) (2023)

## Built on by (most cited first, 7 of 7)

- [Diffusion Models for Future Networks and Communications: A Comprehensive Survey](https://mlsanity.com/paper/2508.01586/) (2025)
- [A Survey on Semantic Communications in Internet of Vehicles](https://mlsanity.com/paper/2503.03767/) (2025)
- [From Large AI Models to Agentic AI: A Tutorial on Future Intelligent Communications](https://mlsanity.com/paper/2505.22311/) (2025)
- [Conquering High Packet-Loss Erasure: MoE Swin Transformer-Based Video Semantic Communication](https://mlsanity.com/paper/2508.01205/) (2025)
- [Generative AI Meets 6G and Beyond: Diffusion Models for Semantic Communications](https://mlsanity.com/paper/2511.08416/) (2025)
- [Optimally Bridging Semantics and Data: Generative Semantic Communication via Schr"odinger Bridge](https://mlsanity.com/paper/2604.17802/) (2026)
- [From Semantic to Token Communication: The Next Paradigm for Large-Model-Driven 6G Intelligent Connectivity](https://mlsanity.com/paper/2609.10714/) (2026)

## Abstract (arXiv)

In the era of 6G, with compelling visions of intelligent transportation systems and digital twins, remote surveillance is poised to become a ubiquitous practice. Substantial data volume and frequent updates present challenges in wireless networks. To address these challenges, we propose a novel agent-driven generative semantic communication (A-GSC) framework based on reinforcement learning. In contrast to the existing research on semantic communication (SemCom), which mainly focuses on either semantic extraction or semantic sampling, we seamlessly integrate both by jointly considering the intrinsic attributes of source information and the contextual information regarding the task. Notably, the introduction of generative artificial intelligence (GAI) enables the independent design of semantic encoders and decoders. In this work, we develop an agent-assisted semantic encoder with cross-modality capability, which can track the semantic changes, channel condition, to perform adaptive semantic extraction and sampling. Accordingly, we design a semantic decoder with both predictive and generative capabilities, consisting of two tailored modules. Moreover, the effectiveness of the designed models has been verified using the UA-DETRAC dataset, demonstrating the performance gains of the overall A-GSC framework in both energy saving and reconstruction accuracy.