# Efficient Self-Ensemble for Semantic Segmentation

arXiv: https://arxiv.org/abs/2111.13280 · HTML: https://mlsanity.com/paper/2111.13280/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 63 references that are in the map.
- First posted on arXiv 26 Nov 2021.

## Builds on (most cited first, 30 of 50)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [Path Aggregation Network for Instance Segmentation](https://mlsanity.com/paper/1803.01534/) (2018)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)

## Built on by (most cited first, 16 of 16)

- [UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformers](https://mlsanity.com/paper/2301.13741/) (2023)
- [Context-Guided Spatial Feature Reconstruction for Efficient Semantic Segmentation](https://mlsanity.com/paper/2405.06228/) (2024)
- [Certified Defences Against Adversarial Patch Attacks on Semantic Segmentation](https://mlsanity.com/paper/2209.05980/) (2022)
- [Embedding-Free Transformer with Inference Spatial Reduction for Efficient Semantic Segmentation](https://mlsanity.com/paper/2407.17261/) (2024)
- [EDTformer: An Efficient Decoder Transformer for Visual Place Recognition](https://mlsanity.com/paper/2412.00784/) (2024)
- [Sub-token ViT Embedding via Stochastic Resonance Transformers](https://mlsanity.com/paper/2310.03967/) (2023)
- [A Convolutional Vision Transformer for Semantic Segmentation of Side-Scan Sonar Data](https://mlsanity.com/paper/2302.12416/) (2023)
- [Revisiting Efficient Semantic Segmentation: Learning Offsets for Better Spatial and Class Feature Alignment](https://mlsanity.com/paper/2508.08811/) (2025)
- [Differentiable Hierarchical Visual Tokenization](https://mlsanity.com/paper/2511.02652/) (2025)
- [Category Feature Transformer for Semantic Segmentation](https://mlsanity.com/paper/2308.05581/) (2023)
- [High-level Feature Guided Decoding for Semantic Segmentation](https://mlsanity.com/paper/2303.08646/) (2023)
- [MaskInversion: Localized Embeddings via Optimization of Explainability Maps](https://mlsanity.com/paper/2407.20034/) (2024)
- [VPNeXt -- Rethinking Dense Decoding for Plain Vision Transformer](https://mlsanity.com/paper/2502.16654/) (2025)
- [More than the Sum of Its Parts: Ensembling Backbone Networks for Few-Shot Segmentation](https://mlsanity.com/paper/2402.06581/) (2024)
- [A Decoding Scheme with Successive Aggregation of Multi-Level Features for Light-Weight Semantic Segmentation](https://mlsanity.com/paper/2402.11201/) (2024)
- [Chanel-Orderer: A Channel-Ordering Predictor for Tri-Channel Natural Images](https://mlsanity.com/paper/2411.13021/) (2024)

## Abstract (arXiv)

Ensemble of predictions is known to perform better than individual predictions taken separately. However, for tasks that require heavy computational resources, e.g. semantic segmentation, creating an ensemble of learners that needs to be trained separately is hardly tractable. In this work, we propose to leverage the performance boost offered by ensemble methods to enhance the semantic segmentation, while avoiding the traditional heavy training cost of the ensemble. Our self-ensemble approach takes advantage of the multi-scale features set produced by feature pyramid network methods to feed independent decoders, thus creating an ensemble within a single model. Similar to the ensemble, the final prediction is the aggregation of the prediction made by each learner. In contrast to previous works, our model can be trained end-to-end, alleviating the traditional cumbersome multi-stage training of ensembles. Our self-ensemble approach outperforms the current state-of-the-art on the benchmark datasets Pascal Context and COCO-Stuff-10K for semantic segmentation and is competitive on ADE20K and Cityscapes. Code is publicly available at github.com/WalBouss/SenFormer.