# Multi-Modal Self-Supervised Semantic Communication

arXiv: https://arxiv.org/abs/2503.13940 · HTML: https://mlsanity.com/paper/2503.13940/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 16 references that are in the map.
- First posted on arXiv 18 Mar 2025.

## Builds on (most cited first, 11 of 11)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Deep Learning Enabled Semantic Communication Systems](https://mlsanity.com/paper/2006.10685/) (2020)
- [Trusted Multi-View Classification with Dynamic Evidential Fusion](https://mlsanity.com/paper/2204.11423/) (2022)
- [Task-Oriented Multi-User Semantic Communications](https://mlsanity.com/paper/2112.10255/) (2021)
- [Semantic-preserved Communication System for Highly Efficient Speech Transmission](https://mlsanity.com/paper/2205.12727/) (2022)
- [Performance Optimization for Semantic Communications: An Attention-based Reinforcement Learning Approach](https://mlsanity.com/paper/2208.08239/) (2022)
- [Tackling Distribution Shifts in Task-Oriented Communication with Information Bottleneck](https://mlsanity.com/paper/2405.09514/) (2024)
- [Deep Joint Source-Channel Coding for Multi-Task Network](https://mlsanity.com/paper/2109.05779/) (2021)
- [Remote Training in Task-Oriented Communication: Supervised or Self-Supervised with Fine-Tuning?](https://mlsanity.com/paper/2502.17922/) (2025)

## Built on by (most cited first, 1 of 1)

- [Communication-Efficient Multi-Modal Edge Inference via Uncertainty-Aware Distributed Learning](https://mlsanity.com/paper/2601.14942/) (2026)

## Abstract (arXiv)

Semantic communication is emerging as a promising paradigm that focuses on the extraction and transmission of semantic meanings using deep learning techniques. While current research primarily addresses the reduction of semantic communication overhead, it often overlooks the training phase, which can incur significant communication costs in dynamic wireless environments. To address this challenge, we propose a multi-modal semantic communication system that leverages multi-modal self-supervised learning to enhance task-agnostic feature extraction. The proposed approach employs self-supervised learning during the pre-training phase to extract task-agnostic semantic features, followed by supervised fine-tuning for downstream tasks. This dual-phase strategy effectively captures both modality-invariant and modality-specific features while minimizing training-related communication overhead. Experimental results on the NYU Depth V2 dataset demonstrate that the proposed method significantly reduces training-related communication overhead while maintaining or exceeding the performance of existing supervised learning approaches. The findings underscore the advantages of multi-modal self-supervised learning in semantic communication, paving the way for more efficient and scalable edge inference systems.