# Nonlinear Transform Source-Channel Coding for Semantic Communications

arXiv: https://arxiv.org/abs/2112.10961 · HTML: https://mlsanity.com/paper/2112.10961/

## Key facts

- Cited by 76 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 47 references that are in the map.
- First posted on arXiv 21 Dec 2021.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018)
- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [High-Fidelity Generative Image Compression](https://mlsanity.com/paper/2006.09965/) (2020)
- [Deep Contextual Video Compression](https://mlsanity.com/paper/2109.15047/) (2021)
- [Nonlinear Transform Coding](https://mlsanity.com/paper/2007.03034/) (2020)
- [DeepJSCC-f: Deep Joint Source-Channel Coding of Images with Feedback](https://mlsanity.com/paper/1911.11174/) (2019)
- [Deep Learning for Joint Source-Channel Coding of Text](https://mlsanity.com/paper/1802.06832/) (2018)
- [Wireless Image Retrieval at the Edge](https://mlsanity.com/paper/2007.10915/) (2020)
- [Bandwidth-Agile Image Transmission with Deep Joint Source-Channel Coding](https://mlsanity.com/paper/2009.12480/) (2020)
- [Deep Joint Source-Channel Coding for Wireless Image Transmission](https://mlsanity.com/paper/1809.01733/) (2018)

## Built on by (most cited first, 30 of 76)

- [Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications](https://mlsanity.com/paper/2207.09353/) (2022)
- [Less Data, More Knowledge: Building Next Generation Semantic Communication Networks](https://mlsanity.com/paper/2211.14343/) (2022)
- [CDDM: Channel Denoising Diffusion Models for Wireless Semantic Communications](https://mlsanity.com/paper/2309.08895/) (2023)
- [Wireless Deep Video Semantic Transmission](https://mlsanity.com/paper/2205.13129/) (2022)
- [DeepJSCC-Q: Constellation Constrained Deep Joint Source-Channel Coding](https://mlsanity.com/paper/2206.08100/) (2022)
- [Joint Source-Channel Coding: Fundamentals and Recent Progress in Practical Designs](https://mlsanity.com/paper/2409.17557/) (2024)
- [Robust Information Bottleneck for Task-Oriented Communication with Digital Modulation](https://mlsanity.com/paper/2209.10382/) (2022)
- [Joint Task and Data Oriented Semantic Communications: A Deep Separate Source-channel Coding Scheme](https://mlsanity.com/paper/2302.13580/) (2023)
- [Diffusion-Driven Semantic Communication for Generative Models with Bandwidth Constraints](https://mlsanity.com/paper/2407.18468/) (2024)
- [Machine Learning for Large-Scale Optimization in 6G Wireless Networks](https://mlsanity.com/paper/2301.03377/) (2023)
- [Generative AI for Physical Layer Communications: A Survey](https://mlsanity.com/paper/2312.05594/) (2023)
- [Semantics-Guided Diffusion for Deep Joint Source-Channel Coding in Wireless Image Transmission](https://mlsanity.com/paper/2501.01138/) (2025)
- [Adaptive Semantic Token Communication for Transformer-based Edge Inference](https://mlsanity.com/paper/2505.17604/) (2025)
- [Latent Diffusion Model-Enabled Low-Latency Semantic Communication in the Presence of Semantic Ambiguities and Wireless Channel Noises](https://mlsanity.com/paper/2406.06644/) (2024)
- [Perceptual Learned Source-Channel Coding for High-Fidelity Image Semantic Transmission](https://mlsanity.com/paper/2205.13120/) (2022)
- [Visual Language Model based Cross-modal Semantic Communication Systems](https://mlsanity.com/paper/2407.00020/) (2024)
- [Diffusion-Aided Joint Source Channel Coding For High Realism Wireless Image Transmission](https://mlsanity.com/paper/2404.17736/) (2024)
- [Adaptive Semantic Token Selection for AI-native Goal-oriented Communications](https://mlsanity.com/paper/2405.02330/) (2024)
- [Robust Image Semantic Coding with Learnable CSI Fusion Masking over MIMO Fading Channels](https://mlsanity.com/paper/2406.07389/) (2024)
- [Generative Video Semantic Communication via Multimodal Semantic Fusion with Large Model](https://mlsanity.com/paper/2502.13838/) (2025)
- [Toward Adaptive Semantic Communications: Efficient Data Transmission via Online Learned Nonlinear Transform Source-Channel Coding](https://mlsanity.com/paper/2211.04339/) (2022)
- [The Larger the Merrier? Efficient Large AI Model Inference in Wireless Edge Networks](https://mlsanity.com/paper/2505.09214/) (2025)
- [Non-Orthogonal Multiple Access Enhanced Multi-User Semantic Communication](https://mlsanity.com/paper/2303.06597/) (2023)
- [How to Evaluate Semantic Communications for Images with ViTScore Metric?](https://mlsanity.com/paper/2309.04891/) (2023)
- [Deep Learning in Physical Layer: Review on Data Driven End-to-End Communication Systems and their Enabling Semantic Applications](https://mlsanity.com/paper/2401.12800/) (2024)
- [MOC-RVQ: Multilevel Codebook-Assisted Digital Generative Semantic Communication](https://mlsanity.com/paper/2401.01272/) (2024)
- [Semantic Edge Computing and Semantic Communications in 6G Networks: A Unifying Survey and Research Challenges](https://mlsanity.com/paper/2411.18199/) (2024)
- [Latent Space Alignment for AI-Native MIMO Semantic Communications](https://mlsanity.com/paper/2507.16680/) (2025)
- [Distortion Resilience for Goal-Oriented Semantic Communication](https://mlsanity.com/paper/2309.14587/) (2023)
- [Deep Joint Semantic Coding and Beamforming for Near-Space Airship-Borne Massive MIMO Network](https://mlsanity.com/paper/2405.19889/) (2024)

## Abstract (arXiv)

In this paper, we propose a class of high-efficiency deep joint source-channel coding methods that can closely adapt to the source distribution under the nonlinear transform, it can be collected under the name nonlinear transform source-channel coding (NTSCC). In the considered model, the transmitter first learns a nonlinear analysis transform to map the source data into latent space, then transmits the latent representation to the receiver via deep joint source-channel coding. Our model incorporates the nonlinear transform as a strong prior to effectively extract the source semantic features and provide side information for source-channel coding. Unlike existing conventional deep joint source-channel coding methods, the proposed NTSCC essentially learns both the source latent representation and an entropy model as the prior on the latent representation. Accordingly, novel adaptive rate transmission and hyperprior-aided codec refinement mechanisms are developed to upgrade deep joint source-channel coding. The whole system design is formulated as an optimization problem whose goal is to minimize the end-to-end transmission rate-distortion performance under established perceptual quality metrics. Across test image sources with various resolutions, we find that the proposed NTSCC transmission method generally outperforms both the analog transmission using the standard deep joint source-channel coding and the classical separation-based digital transmission. Notably, the proposed NTSCC method can potentially support future semantic communications due to its content-aware ability and perceptual optimization goal.