# Robust Semantic Communications with Masked VQ-VAE Enabled Codebook

arXiv: https://arxiv.org/abs/2206.04011 · HTML: https://mlsanity.com/paper/2206.04011/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 44 references that are in the map.
- First posted on arXiv 8 Jun 2022.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [The Limitations of Deep Learning in Adversarial Settings](https://mlsanity.com/paper/1511.07528/) (2015)
- [Universal adversarial perturbations](https://mlsanity.com/paper/1610.08401/) (2016)
- [Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks](https://mlsanity.com/paper/1511.04508/) (2015)
- [Defense against Adversarial Attacks Using High-Level Representation Guided Denoiser](https://mlsanity.com/paper/1712.02976/) (2017)
- [Deep Learning Enabled Semantic Communication Systems](https://mlsanity.com/paper/2006.10685/) (2020)
- [An Introduction to Deep Learning for the Physical Layer](https://mlsanity.com/paper/1702.00832/) (2017)
- [The Roadmap to 6G -- AI Empowered Wireless Networks](https://mlsanity.com/paper/1904.11686/) (2019)
- [Towards Deep Neural Network Architectures Robust to Adversarial Examples](https://mlsanity.com/paper/1412.5068/) (2014)
- [Learning Task-Oriented Communication for Edge Inference: An Information Bottleneck Approach](https://mlsanity.com/paper/2102.04170/) (2021)
- [Semantic Communication Systems for Speech Transmission](https://mlsanity.com/paper/2102.12605/) (2021)
- [DeepJSCC-f: Deep Joint Source-Channel Coding of Images with Feedback](https://mlsanity.com/paper/1911.11174/) (2019)
- [Semantic Communications: Principles and Challenges](https://mlsanity.com/paper/2201.01389/) (2021)
- [Improving Adversarial Robustness via Channel-wise Activation Suppressing](https://mlsanity.com/paper/2103.08307/) (2021)
- [Task-Oriented Multi-User Semantic Communications](https://mlsanity.com/paper/2112.10255/) (2021)
- [Semantics-Empowered Communication for Networked Intelligent Systems](https://mlsanity.com/paper/2007.11579/) (2020)
- [Deep Learning for IoT Big Data and Streaming Analytics: A Survey](https://mlsanity.com/paper/1712.04301/) (2017)
- [Wireless Image Retrieval at the Edge](https://mlsanity.com/paper/2007.10915/) (2020)
- [From Semantic Communication to Semantic-aware Networking: Model, Architecture, and Open Problems](https://mlsanity.com/paper/2012.15405/) (2020)
- [Deep Source-Channel Coding for Sentence Semantic Transmission with HARQ](https://mlsanity.com/paper/2106.03009/) (2021)
- [Channel-Aware Adversarial Attacks Against Deep Learning-Based Wireless Signal Classifiers](https://mlsanity.com/paper/2005.05321/) (2020)
- [Reinforcement Learning-powered Semantic Communication via Semantic Similarity](https://mlsanity.com/paper/2108.12121/) (2021)
- [Towards Goal-Oriented Semantic Signal Processing: Applications and Future Challenges](https://mlsanity.com/paper/2109.11885/) (2021)

## Built on by (most cited first, 30 of 56)

- [PastNet: Introducing Physical Inductive Biases for Spatio-temporal Video Prediction](https://mlsanity.com/paper/2305.11421/) (2023)
- [StrokeNUWA: Tokenizing Strokes for Vector Graphic Synthesis](https://mlsanity.com/paper/2401.17093/) (2024)
- [Zero-Knowledge Proof-based Verifiable Decentralized Machine Learning in Communication Network: A Comprehensive Survey](https://mlsanity.com/paper/2310.14848/) (2023)
- [Vector Quantized Semantic Communication System](https://mlsanity.com/paper/2209.11519/) (2022)
- [OFDM-Based Digital Semantic Communication with Importance Awareness](https://mlsanity.com/paper/2401.02178/) (2024)
- [Latent Semantic Diffusion-based Channel Adaptive De-Noising SemCom for Future 6G Systems](https://mlsanity.com/paper/2304.09420/) (2023)
- [Knowledge Base Enabled Semantic Communication: A Generative Perspective](https://mlsanity.com/paper/2311.12443/) (2023)
- [Adaptive Resource Allocation for Semantic Communication Networks](https://mlsanity.com/paper/2312.01081/) (2023)
- [Applications of Generative AI (GAI) for Mobile and Wireless Networking: A Survey](https://mlsanity.com/paper/2405.20024/) (2024)
- [Robust Image Semantic Coding with Learnable CSI Fusion Masking over MIMO Fading Channels](https://mlsanity.com/paper/2406.07389/) (2024)
- [Semantic Successive Refinement: A Generative AI-aided Semantic Communication Framework](https://mlsanity.com/paper/2408.05112/) (2024)
- [Adversarial Attacks and Defenses for Semantic Communication in Vehicular Metaverses](https://mlsanity.com/paper/2306.03528/) (2023)
- [Synchronous Multi-modal Semantic Communication System with Packet-level Coding](https://mlsanity.com/paper/2408.04535/) (2024)
- [Scene123: One Prompt to 3D Scene Generation via Video-Assisted and Consistency-Enhanced MAE](https://mlsanity.com/paper/2408.05477/) (2024)
- [Performance Limits of a Deep Learning-Enabled Text Semantic Communication under Interference](https://mlsanity.com/paper/2302.14702/) (2023)
- [Non-Orthogonal Multiple Access Enhanced Multi-User Semantic Communication](https://mlsanity.com/paper/2303.06597/) (2023)
- [Rate-Adaptive Coding Mechanism for Semantic Communications With Multi-Modal Data](https://mlsanity.com/paper/2305.10773/) (2023)
- [How to Evaluate Semantic Communications for Images with ViTScore Metric?](https://mlsanity.com/paper/2309.04891/) (2023)
- [Vision Transformer-based Semantic Communications With Importance-Aware Quantization](https://mlsanity.com/paper/2412.06038/) (2024)
- [Communication-Efficient Framework for Distributed Image Semantic Wireless Transmission](https://mlsanity.com/paper/2308.03713/) (2023)
- [Diff-GO: Diffusion Goal-Oriented Communications to Achieve Ultra-High Spectrum Efficiency](https://mlsanity.com/paper/2312.02984/) (2023)
- [Codebook-enabled Generative End-to-end Semantic Communication Powered by Transformer](https://mlsanity.com/paper/2402.16868/) (2024)
- [Learned Image Transmission with Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2408.16340/) (2024)
- [Task-Oriented Communication for Graph Data: A Graph Information Bottleneck Approach](https://mlsanity.com/paper/2409.02728/) (2024)
- [Toward Mixture-of-Experts Enabled Trustworthy Semantic Communication for 6G Networks](https://mlsanity.com/paper/2409.15695/) (2024)
- [Generative Semantic Communication for Text-to-Speech Synthesis](https://mlsanity.com/paper/2410.03459/) (2024)
- [Resource-Efficient Multiview Perception: Integrating Semantic Masking with Masked Autoencoders](https://mlsanity.com/paper/2410.04817/) (2024)
- [RIS-aided Latent Space Alignment for Semantic Channel Equalization](https://mlsanity.com/paper/2507.16450/) (2025)
- [Magmaw: Modality-Agnostic Adversarial Attacks on Machine Learning-Based Wireless Communication Systems](https://mlsanity.com/paper/2311.00207/) (2023)
- [DRL Optimization Trajectory Generation via Wireless Network Intent-Guided Diffusion Models for Optimizing Resource Allocation](https://mlsanity.com/paper/2410.14481/) (2024)

## Abstract (arXiv)

Although semantic communications have exhibited satisfactory performance for a large number of tasks, the impact of semantic noise and the robustness of the systems have not been well investigated. Semantic noise refers to the misleading between the intended semantic symbols and received ones, thus cause the failure of tasks. In this paper, we first propose a framework for the robust end-to-end semantic communication systems to combat the semantic noise. In particular, we analyze sample-dependent and sample-independent semantic noise. To combat the semantic noise, the adversarial training with weight perturbation is developed to incorporate the samples with semantic noise in the training dataset. Then, we propose to mask a portion of the input, where the semantic noise appears frequently, and design the masked vector quantized-variational autoencoder (VQ-VAE) with the noise-related masking strategy. We use a discrete codebook shared by the transmitter and the receiver for encoded feature representation. To further improve the system robustness, we develop a feature importance module (FIM) to suppress the noise-related and task-unrelated features. Thus, the transmitter simply needs to transmit the indices of these important task-related features in the codebook. Simulation results show that the proposed method can be applied in many downstream tasks and significantly improve the robustness against semantic noise with remarkable reduction on the transmission overhead.