# On the Role of ViT and CNN in Semantic Communications: Analysis and Prototype Validation

arXiv: https://arxiv.org/abs/2306.02759 · HTML: https://mlsanity.com/paper/2306.02759/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jun 2023.

## Built on by (most cited first, 8 of 8)

- [Importance-Aware Image Segmentation-based Semantic Communication for Autonomous Driving](https://mlsanity.com/paper/2401.10153/) (2024)
- [Attention-aware Semantic Communications for Collaborative Inference](https://mlsanity.com/paper/2404.07217/) (2024)
- [Deep Learning in Physical Layer: Review on Data Driven End-to-End Communication Systems and their Enabling Semantic Applications](https://mlsanity.com/paper/2401.12800/) (2024)
- [On-Air Deep Learning Integrated Semantic Inference Models for Enhanced Earth Observation Satellite Networks](https://mlsanity.com/paper/2409.15246/) (2024)
- [Semantic Edge-Cloud Communication for Real-Time Urban Traffic Surveillance with ViT and LLMs over Mobile Networks](https://mlsanity.com/paper/2509.21259/) (2025)
- [Hybrid Bit and Semantic Communications](https://mlsanity.com/paper/2404.19477/) (2024)
- [Object-Attribute-Relation Representation Based Video Semantic Communication](https://mlsanity.com/paper/2406.10469/) (2024)
- [Bridging Neural Networks and Wireless Systems with MIMO-OFDM Semantic Communications](https://mlsanity.com/paper/2501.16726/) (2025)

## Abstract (arXiv)

Semantic communications have shown promising advancements by optimizing source and channel coding jointly. However, the dynamics of these systems remain understudied, limiting research and performance gains. Inspired by the robustness of Vision Transformers (ViTs) in handling image nuisances, we propose a ViT-based model for semantic communications. Our approach achieves a peak signal-to-noise ratio (PSNR) gain of +0.5 dB over convolutional neural network variants. We introduce novel measures, average cosine similarity and Fourier analysis, to analyze the inner workings of semantic communications and optimize the system's performance. We also validate our approach through a real wireless channel prototype using software-defined radio (SDR). To the best of our knowledge, this is the first investigation of the fundamental workings of a semantic communications system, accompanied by the pioneering hardware implementation. To facilitate reproducibility and encourage further research, we provide open-source code, including neural network implementations and LabVIEW codes for SDR-based wireless transmission systems.