# SCTNet: Single-Branch CNN with Transformer Semantic Information for Real-Time Segmentation

arXiv: https://arxiv.org/abs/2312.17071 · HTML: https://mlsanity.com/paper/2312.17071/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 50 references that are in the map.
- First posted on arXiv 28 Dec 2023.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [Twins: Revisiting the Design of Spatial Attention in Vision Transformers](https://mlsanity.com/paper/2104.13840/) (2021)
- [Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs](https://mlsanity.com/paper/1412.7062/) (2014)
- [RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/1611.06612/) (2016)
- [BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentation](https://mlsanity.com/paper/2004.02147/) (2020)
- [ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation](https://mlsanity.com/paper/1606.02147/) (2016)
- [ICNet for Real-Time Semantic Segmentation on High-Resolution Images](https://mlsanity.com/paper/1704.08545/) (2017)
- [SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation](https://mlsanity.com/paper/2209.08575/) (2022)
- [Channel-wise Knowledge Distillation for Dense Prediction](https://mlsanity.com/paper/2011.13256/) (2020)
- [OCNet: Object Context Network for Scene Parsing](https://mlsanity.com/paper/1809.00916/) (2018)

## Built on by (most cited first, 22 of 22)

- [CaRe-Ego: Contact-aware Relationship Modeling for Egocentric Interactive Hand-object Segmentation](https://mlsanity.com/paper/2407.05576/) (2024)
- [DSNet: A Novel Way to Use Atrous Convolutions in Semantic Segmentation](https://mlsanity.com/paper/2406.03702/) (2024)
- [CTA-Net: A CNN-Transformer Aggregation Network for Improving Multi-Scale Feature Extraction](https://mlsanity.com/paper/2410.11428/) (2024)
- [Golden Cudgel Network for Real-Time Semantic Segmentation](https://mlsanity.com/paper/2503.03325/) (2025)
- [Interaction-aware Representation Modeling with Co-occurrence Consistency for Egocentric Hand-Object Parsing](https://mlsanity.com/paper/2602.20597/) (2026)
- [MacFormer: Semantic Segmentation with Fine Object Boundaries](https://mlsanity.com/paper/2408.05699/) (2024)
- [A Deep Semantic Segmentation Network with Semantic and Contextual Refinements](https://mlsanity.com/paper/2412.08671/) (2024)
- [Temporal-Spatial Decouple before Act: Disentangled Representation Learning for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2601.13659/) (2026)
- [MarsSeg: Mars Surface Semantic Segmentation with Multi-level Extractor and Connector](https://mlsanity.com/paper/2404.04155/) (2024)
- [SCASeg: Strip Cross-Attention for Efficient Semantic Segmentation](https://mlsanity.com/paper/2411.17061/) (2024)
- [A Semantic Communication System for Real-time 3D Reconstruction Tasks](https://mlsanity.com/paper/2412.01191/) (2024)
- [Structural Entropy Guided Probabilistic Coding](https://mlsanity.com/paper/2412.08841/) (2024)
- [Vehicular Road Crack Detection with Deep Learning: A New Online Benchmark for Comprehensive Evaluation of Existing Algorithms](https://mlsanity.com/paper/2503.18082/) (2025)
- [ECMNet:Lightweight Semantic Segmentation with Efficient CNN-Mamba Network](https://mlsanity.com/paper/2506.08629/) (2025)
- [ProtoSolo: Interpretable Image Classification via Single-Prototype Activation](https://mlsanity.com/paper/2506.19808/) (2025)
- [EfficientCrackNet: A Lightweight Model for Crack Segmentation](https://mlsanity.com/paper/2409.18099/) (2024)
- [MSV-Mamba: A Multiscale Vision Mamba Network for Echocardiography Segmentation](https://mlsanity.com/paper/2501.07120/) (2025)
- [ChatStitch: Visualizing Through Structures via Surround-View Unsupervised Deep Image Stitching with Collaborative LLM-Agents](https://mlsanity.com/paper/2503.14948/) (2025)
- [A Novel Attention-Augmented Wavelet YOLO System for Real-time Brain Vessel Segmentation on Transcranial Color-coded Doppler](https://mlsanity.com/paper/2508.13875/) (2025)
- [EPARA: Parallelizing Categorized AI Inference in Edge Clouds](https://mlsanity.com/paper/2511.00603/) (2025)
- [GeoIB: Geometry-Aware Information Bottleneck via Statistical-Manifold Compression](https://mlsanity.com/paper/2602.03906/) (2026)
- [State Space Models Meet Remote Sensing: A Survey](https://mlsanity.com/paper/2606.25329/) (2026)

## Abstract (arXiv)

Recent real-time semantic segmentation methods usually adopt an additional semantic branch to pursue rich long-range context. However, the additional branch incurs undesirable computational overhead and slows inference speed. To eliminate this dilemma, we propose SCTNet, a single branch CNN with transformer semantic information for real-time segmentation. SCTNet enjoys the rich semantic representations of an inference-free semantic branch while retaining the high efficiency of lightweight single branch CNN. SCTNet utilizes a transformer as the training-only semantic branch considering its superb ability to extract long-range context. With the help of the proposed transformer-like CNN block CFBlock and the semantic information alignment module, SCTNet could capture the rich semantic information from the transformer branch in training. During the inference, only the single branch CNN needs to be deployed. We conduct extensive experiments on Cityscapes, ADE20K, and COCO-Stuff-10K, and the results show that our method achieves the new state-of-the-art performance. The code and model is available at https://github.com/xzz777/SCTNet