# Large-Scale Model Enabled Semantic Communication Based on Robust Knowledge Distillation

arXiv: https://arxiv.org/abs/2508.02148 · HTML: https://mlsanity.com/paper/2508.02148/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 27 of its 35 references that are in the map.
- First posted on arXiv 4 Aug 2025.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Deep Learning Enabled Semantic Communication Systems](https://mlsanity.com/paper/2006.10685/) (2020)
- [Semantic Communication Systems for Speech Transmission](https://mlsanity.com/paper/2102.12605/) (2021)
- [Task-Oriented Multi-User Semantic Communications](https://mlsanity.com/paper/2112.10255/) (2021)
- [Deep Learning for Joint Source-Channel Coding of Text](https://mlsanity.com/paper/1802.06832/) (2018)
- [Less Data, More Knowledge: Building Next Generation Semantic Communication Networks](https://mlsanity.com/paper/2211.14343/) (2022)
- [A Unified Multi-Task Semantic Communication System for Multimodal Data](https://mlsanity.com/paper/2209.07689/) (2022)
- [Artificial General Intelligence (AGI)-Native Wireless Systems: A Journey Beyond 6G](https://mlsanity.com/paper/2405.02336/) (2024)
- [Joint Source-Channel Coding for Channel-Adaptive Digital Semantic Communications](https://mlsanity.com/paper/2311.08146/) (2023)
- [OFDM-Based Digital Semantic Communication with Importance Awareness](https://mlsanity.com/paper/2401.02178/) (2024)
- [Large-Language-Model Enabled Semantic Communication Systems](https://mlsanity.com/paper/2407.14112/) (2024)
- [Large-Scale AI in Telecom: Charting the Roadmap for Innovation, Scalability, and Enhanced Digital Experiences](https://mlsanity.com/paper/2503.04184/) (2025)
- [Demo: Real-Time Semantic Communications with a Vision Transformer](https://mlsanity.com/paper/2205.03886/) (2022)
- [Semantic Importance-Aware Communications with Semantic Correction Using Large Language Models](https://mlsanity.com/paper/2405.16011/) (2024)
- [Rethinking Generative Semantic Communication for Multi-User Systems with Large Language Models](https://mlsanity.com/paper/2408.08765/) (2024)
- [Adaptive Information Bottleneck Guided Joint Source and Channel Coding for Image Transmission](https://mlsanity.com/paper/2203.06492/) (2022)
- [Large Language Model-Based Semantic Communication System for Image Transmission](https://mlsanity.com/paper/2501.12988/) (2025)
- [Generative AI Driven Task-Oriented Adaptive Semantic Communications](https://mlsanity.com/paper/2407.11354/) (2024)
- [Explainable Semantic Communication for Text Tasks](https://mlsanity.com/paper/2303.12286/) (2023)
- [Scene Understanding Enabled Semantic Communication with Open Channel Coding](https://mlsanity.com/paper/2501.14520/) (2025)

## Abstract (arXiv)

Large-scale models (LSMs) can be an effective framework for semantic representation and understanding, thereby providing a suitable tool for designing semantic communication (SC) systems. However, their direct deployment is often hindered by high computational complexity and resource requirements. In this paper, a novel robust knowledge distillation based semantic communication (RKD-SC) framework is proposed to enable efficient and textcolorblackchannel-noise-robust LSM-powered SC. The framework addresses two key challenges: determining optimal compact model architectures and effectively transferring knowledge while maintaining robustness against channel noise. First, a knowledge distillation-based lightweight differentiable architecture search (KDL-DARTS) algorithm is proposed. This algorithm integrates knowledge distillation loss and a complexity penalty into the neural architecture search process to identify high-performance, lightweight semantic encoder architectures. Second, a novel two-stage robust knowledge distillation (RKD) algorithm is developed to transfer semantic capabilities from an LSM (teacher) to a compact encoder (student) and subsequently enhance system robustness. To further improve resilience to channel impairments, a channel-aware transformer (CAT) block is introduced as the channel codec, trained under diverse channel conditions with variable-length outputs. Extensive simulations on image classification tasks demonstrate that the RKD-SC framework significantly reduces model parameters while preserving a high degree of the teacher model's performance and exhibiting superior robustness compared to existing methods.