# Reinforcement Learning-powered Semantic Communication via Semantic Similarity

arXiv: https://arxiv.org/abs/2108.12121 · HTML: https://mlsanity.com/paper/2108.12121/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 76 references that are in the map.
- First posted on arXiv 27 Aug 2021.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient](https://mlsanity.com/paper/1609.05473/) (2016)
- [MagFace: A Universal Representation for Face Recognition and Quality Assessment](https://mlsanity.com/paper/2103.06627/) (2021)
- [Deep Learning Enabled Semantic Communication Systems](https://mlsanity.com/paper/2006.10685/) (2020)
- [An Introduction to Deep Learning for the Physical Layer](https://mlsanity.com/paper/1702.00832/) (2017)
- [Applications of Deep Reinforcement Learning in Communications and Networking: A Survey](https://mlsanity.com/paper/1810.07862/) (2018)
- [An Actor-Critic Algorithm for Sequence Prediction](https://mlsanity.com/paper/1607.07086/) (2016)
- [Improved Image Captioning via Policy Gradient optimization of SPIDEr](https://mlsanity.com/paper/1612.00370/) (2016)
- [Semantics-aware BERT for Language Understanding](https://mlsanity.com/paper/1909.02209/) (2019)
- [6G Networks: Beyond Shannon Towards Semantic and Goal-Oriented Communications](https://mlsanity.com/paper/2011.14844/) (2020)
- [BottleNet++: An End-to-End Approach for Feature Compression in Device-Edge Co-Inference Systems](https://mlsanity.com/paper/1910.14315/) (2019)
- [Evolution of Semantic Similarity -- A Survey](https://mlsanity.com/paper/2004.13820/) (2020)
- [On the Sentence Embeddings from Pre-trained Language Models](https://mlsanity.com/paper/2011.05864/) (2020)
- [A Lite Distributed Semantic Communication System for Internet of Things](https://mlsanity.com/paper/2007.11095/) (2020)
- [Deep Learning for Joint Source-Channel Coding of Text](https://mlsanity.com/paper/1802.06832/) (2018)
- [Deep Reinforcement Learning-based Image Captioning with Embedding Reward](https://mlsanity.com/paper/1704.03899/) (2017)
- [Deep Joint Source-Channel Coding for Wireless Image Transmission](https://mlsanity.com/paper/1809.01733/) (2018)
- [Rethinking the Discount Factor in Reinforcement Learning: A Decision Theoretic Approach](https://mlsanity.com/paper/1902.02893/) (2019)
- [Model-free Training of End-to-end Communication Systems](https://mlsanity.com/paper/1812.05929/) (2018)
- [Deep Source-Channel Coding for Sentence Semantic Transmission with HARQ](https://mlsanity.com/paper/2106.03009/) (2021)
- [Deep Reinforcement Learning for Resource Management in Network Slicing](https://mlsanity.com/paper/1805.06591/) (2018)
- [A Better Variant of Self-Critical Sequence Training](https://mlsanity.com/paper/2003.09971/) (2020)
- [Semantics-Native Communication with Contextual Reasoning](https://mlsanity.com/paper/2108.05681/) (2021)
- [Semantic Communication with Adaptive Universal Transformer](https://mlsanity.com/paper/2108.09119/) (2021)

## Built on by (most cited first, 15 of 15)

- [Robust Semantic Communications with Masked VQ-VAE Enabled Codebook](https://mlsanity.com/paper/2206.04011/) (2022)
- [Towards Semantic Communications: Deep Learning-Based Image Semantic Coding](https://mlsanity.com/paper/2208.04094/) (2022)
- [Curriculum Learning for Goal-Oriented Semantic Communications with a Common Language](https://mlsanity.com/paper/2204.10429/) (2022)
- [Wireless End-to-End Image Transmission System using Semantic Communications](https://mlsanity.com/paper/2302.13721/) (2023)
- [Common Language for Goal-Oriented Semantic Communications: A Curriculum Learning Framework](https://mlsanity.com/paper/2111.08051/) (2021)
- [Demo: Real-Time Semantic Communications with a Vision Transformer](https://mlsanity.com/paper/2205.03886/) (2022)
- [Adaptive Resource Allocation for Semantic Communication Networks](https://mlsanity.com/paper/2312.01081/) (2023)
- [Semantics-Empowered Communication: A Tutorial-cum-Survey](https://mlsanity.com/paper/2212.08487/) (2022)
- [Optimization of Image Transmission in a Cooperative Semantic Communication Networks](https://mlsanity.com/paper/2301.00433/) (2023)
- [Deep Learning in Physical Layer: Review on Data Driven End-to-End Communication Systems and their Enabling Semantic Applications](https://mlsanity.com/paper/2401.12800/) (2024)
- [Towards Ubiquitous Semantic Metaverse: Challenges, Approaches, and Opportunities](https://mlsanity.com/paper/2307.06687/) (2023)
- [Model-free Reinforcement Learning of Semantic Communication by Stochastic Policy Gradient](https://mlsanity.com/paper/2305.03571/) (2023)
- [Alternate Learning based Sparse Semantic Communications for Visual Transmission](https://mlsanity.com/paper/2309.16681/) (2023)
- [Context-Aware Composition of Agent Policies by Markov Decision Process Entity Embeddings and Agent Ensembles](https://mlsanity.com/paper/2308.14521/) (2023)
- [Interplay of Semantic Communication and Knowledge Learning](https://mlsanity.com/paper/2402.03339/) (2024)

## Abstract (arXiv)

We introduce a new semantic communication mechanism - SemanticRL, whose key idea is to preserve the semantic information instead of strictly securing the bit-level precision. Unlike previous methods that mainly concentrate on the network or structure design, we revisit the learning process and point out the semantic blindness of commonly used objective functions. To address this semantic gap, we introduce a schematic shift that learns from semantic similarity, instead of relying on conventional paired bit-level supervisions like cross entropy and bit error rate. However, developing such a semantic communication system is indeed a nontrivial task considering the non-differentiability of most semantic metrics as well as the instability from noisy channels. To further resolve these issues, we put forward a self-critic reinforcement learning (RL) solution which allows an efficient and stable learning on any user-defined semantic measurement, and take a step further to simultaneously tackle the non-differentiable semantic channel optimization problem via self-critic stochastic iterative updating (SCSIU) training on the decoupled semantic transceiver. We have firstly tested the proposed method in the challenging European-parliament dataset, which confirms the superiority of our method in revealing the semantic meanings, and better handling the semantic noise. Apart from the experimental results, we further provide an in-depth look at how the semantic model behaves, along with its superb generalization ability in real-life examples. An RL-based image transmission extension is also exemplified, so as to prove the generalization ability and motivate future discussion.