# Large Generative Model Assisted 3D Semantic Communication arXiv: https://arxiv.org/abs/2403.05783 ยท HTML: https://mlsanity.com/paper/2403.05783/ ## Key facts - Cited by 12 papers in the map (data through 14 Oct 2026). - Builds on 29 of its 50 references that are in the map. - First posted on arXiv 9 Mar 2024. ## Builds on (most cited first, 29 of 29) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023) - [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022) - [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022) - [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022) - [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021) - [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022) - [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021) - [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016) - [GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generation](https://mlsanity.com/paper/2203.02923/) (2022) - [Learning Deconvolution Network for Semantic Segmentation](https://mlsanity.com/paper/1505.04366/) (2015) - [Inpaint Anything: Segment Anything Meets Image Inpainting](https://mlsanity.com/paper/2304.06790/) (2023) - [Image De-raining Using a Conditional Generative Adversarial Network](https://mlsanity.com/paper/1701.05957/) (2017) - [NeRFReN: Neural Radiance Fields with Reflections](https://mlsanity.com/paper/2111.15234/) (2021) - [Can SAM Segment Anything? When SAM Meets Camouflaged Object Detection](https://mlsanity.com/paper/2304.04709/) (2023) - [Semantic Communications for Future Internet: Fundamentals, Applications, and Challenges](https://mlsanity.com/paper/2207.00427/) (2022) - [Semantic Communications: Principles and Challenges](https://mlsanity.com/paper/2201.01389/) (2021) - [A Lite Distributed Semantic Communication System for Internet of Things](https://mlsanity.com/paper/2007.11095/) (2020) - [Loc-NeRF: Monte Carlo Localization using Neural Radiance Fields](https://mlsanity.com/paper/2209.09050/) (2022) - [Segment Anything in 3D with Radiance Fields](https://mlsanity.com/paper/2304.12308/) (2023) - [Large Language Model Enhanced Multi-Agent Systems for 6G Communications](https://mlsanity.com/paper/2312.07850/) (2023) - [Large AI Model Empowered Multimodal Semantic Communications](https://mlsanity.com/paper/2309.01249/) (2023) - [Large AI Model-Based Semantic Communications](https://mlsanity.com/paper/2307.03492/) (2023) - [A Survey on Semantic Communications for Intelligent Wireless Networks](https://mlsanity.com/paper/2202.03705/) (2022) - [Semantic-Aware Sensing Information Transmission for Metaverse: A Contest Theoretic Approach](https://mlsanity.com/paper/2211.12783/) (2022) - [End-to-End Fast Training of Communication Links Without a Channel Model via Online Meta-Learning](https://mlsanity.com/paper/2003.01479/) (2020) - [Channel Estimation for One-Bit Multiuser Massive MIMO Using Conditional GAN](https://mlsanity.com/paper/2006.11435/) (2020) - [Distributed Conditional Generative Adversarial Networks (GANs) for Data-Driven Millimeter Wave Communications in UAV Networks](https://mlsanity.com/paper/2102.01751/) (2021) - [Towards Holographic Video Communications: A Promising AI-driven Solution](https://mlsanity.com/paper/2210.06794/) (2022) ## Built on by (most cited first, 12 of 12) - [Generative Diffusion Models for Wireless Networks: Fundamental, Architecture, and State-of-the-Art](https://mlsanity.com/paper/2507.16733/) (2025) - [Latent Diffusion Model-Enabled Low-Latency Semantic Communication in the Presence of Semantic Ambiguities and Wireless Channel Noises](https://mlsanity.com/paper/2406.06644/) (2024) - [SIMAC: A Semantic-Driven Integrated Multimodal Sensing And Communication Framework](https://mlsanity.com/paper/2503.08726/) (2025) - [Generative Semantic Communication: Architectures, Technologies, and Applications](https://mlsanity.com/paper/2412.08642/) (2024) - [Semantic Communications with Computer Vision Sensing for Edge Video Transmission](https://mlsanity.com/paper/2503.07252/) (2025) - [Large Language Model-Driven Distributed Integrated Multimodal Sensing and Semantic Communications](https://mlsanity.com/paper/2505.18194/) (2025) - [From Large AI Models to Agentic AI: A Tutorial on Future Intelligent Communications](https://mlsanity.com/paper/2505.22311/) (2025) - [Personalized Federated Learning for Generative AI-Assisted Semantic Communications](https://mlsanity.com/paper/2410.02450/) (2024) - [Channel-adaptive Cross-modal Generative Semantic Communication for Point Cloud Transmission](https://mlsanity.com/paper/2506.03211/) (2025) - [GAI-Enabled Explainable Personalized Federated Semi-Supervised Learning](https://mlsanity.com/paper/2410.08634/) (2024) - [Foundation Model-Based Adaptive Semantic Image Transmission for Dynamic Wireless Environments](https://mlsanity.com/paper/2509.23590/) (2025) - [A Survey on Semantic Communication for Vision: Categories, Frameworks, Enabling Techniques, and Applications](https://mlsanity.com/paper/2601.22202/) (2026) ## Abstract (arXiv) Semantic Communication (SC) is a novel paradigm for data transmission in 6G. However, there are several challenges posed when performing SC in 3D scenarios: 1) 3D semantic extraction; 2) Latent semantic redundancy; and 3) Uncertain channel estimation. To address these issues, we propose a Generative AI Model assisted 3D SC (GAM-3DSC) system. Firstly, we introduce a 3D Semantic Extractor (3DSE), which employs generative AI models, including Segment Anything Model (SAM) and Neural Radiance Field (NeRF), to extract key semantics from a 3D scenario based on user requirements. The extracted 3D semantics are represented as multi-perspective images of the goal-oriented 3D object. Then, we present an Adaptive Semantic Compression Model (ASCM) for encoding these multi-perspective images, in which we use a semantic encoder with two output heads to perform semantic encoding and mask redundant semantics in the latent semantic space, respectively. Next, we design a conditional Generative adversarial network and Diffusion model aided-Channel Estimation (GDCE) to estimate and refine the Channel State Information (CSI) of physical channels. Finally, simulation results demonstrate the advantages of the proposed GAM-3DSC system in effectively transmitting the goal-oriented 3D scenario.