# Generative Transformer for Accurate and Reliable Salient Object Detection

arXiv: https://arxiv.org/abs/2104.10127 · HTML: https://mlsanity.com/paper/2104.10127/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 87 of its 186 references that are in the map.
- First posted on arXiv 20 Apr 2021.

## Builds on (most cited first, 30 of 87)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [Provable defenses against adversarial examples via the convex outer adversarial polytope](https://mlsanity.com/paper/1711.00851/) (2017)
- [Understanding the Effective Receptive Field in Deep Convolutional Neural Networks](https://mlsanity.com/paper/1701.04128/) (2017)
- [Structure-measure: A New Way to Evaluate Foreground Maps](https://mlsanity.com/paper/1708.00786/) (2017)
- [Enhanced-alignment Measure for Binary Foreground Map Evaluation](https://mlsanity.com/paper/1805.10421/) (2018)
- [Learning Spatio-Temporal Transformer for Visual Tracking](https://mlsanity.com/paper/2103.17154/) (2021)
- [ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1604.05144/) (2016)
- [Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/1803.10464/) (2018)
- [What's the Point: Semantic Segmentation with Point Supervision](https://mlsanity.com/paper/1506.02106/) (2015)
- [A Probabilistic U-Net for Segmentation of Ambiguous Images](https://mlsanity.com/paper/1806.05034/) (2018)

## Built on by (most cited first, 18 of 18)

- [Multimodal Variational Auto-encoder based Audio-Visual Segmentation](https://mlsanity.com/paper/2310.08303/) (2023)
- [Contrastive Conditional Latent Diffusion for Audio-visual Segmentation](https://mlsanity.com/paper/2307.16579/) (2023)
- [HRTransNet: HRFormer-Driven Two-Modality Salient Object Detection](https://mlsanity.com/paper/2301.03036/) (2023)
- [Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation](https://mlsanity.com/paper/2312.06462/) (2023)
- [Salient Object Detection in Optical Remote Sensing Images Driven by Transformer](https://mlsanity.com/paper/2309.08206/) (2023)
- [Boosting Salient Object Detection with Transformer-based Asymmetric Bilateral U-Net](https://mlsanity.com/paper/2108.07851/) (2021)
- [Mutual Information Regularization for Weakly-supervised RGB-D Salient Object Detection](https://mlsanity.com/paper/2306.03630/) (2023)
- [AVESFormer: Efficient Transformer Design for Real-Time Audio-Visual Segmentation](https://mlsanity.com/paper/2408.01708/) (2024)
- [Deep Learning in Concealed Dense Prediction](https://mlsanity.com/paper/2504.10979/) (2025)
- [Depth-guided Texture Diffusion for Image Semantic Segmentation](https://mlsanity.com/paper/2408.09097/) (2024)
- [Salient Object Detection in Traffic Scene through the TSOD10K Dataset](https://mlsanity.com/paper/2503.16910/) (2025)
- [OpenAVS: Training-Free Open-Vocabulary Audio Visual Segmentation with Foundational Models](https://mlsanity.com/paper/2505.01448/) (2025)
- [A Generative Victim Model for Segmentation](https://mlsanity.com/paper/2412.07274/) (2024)
- [Graph-Based Uncertainty Modeling and Multimodal Fusion for Salient Object Detection](https://mlsanity.com/paper/2508.20415/) (2025)
- [DualGazeNet: A Biologically Inspired Dual-Gaze Query Network for Salient Object Detection](https://mlsanity.com/paper/2511.18865/) (2025)
- [Learning Spatially Adaptive Structural Coordination for Underwater Salient Object Detection](https://mlsanity.com/paper/2605.15535/) (2026)
- [Rethinking Conditional Generation for Underwater Salient Object Detection](https://mlsanity.com/paper/2607.01825/) (2026)
- [Spectral-Spatial Synergistic Guided Network for Hyperspectral Salient Object Detection](https://mlsanity.com/paper/2607.21032/) (2026)

## Abstract (arXiv)

Transformer, which originates from machine translation, is particularly powerful at modeling long-range dependencies. Currently, the transformer is making revolutionary progress in various vision tasks, leading to significant performance improvements compared with the convolutional neural network (CNN) based frameworks. In this paper, we conduct extensive research on exploiting the contributions of transformers for accurate and reliable salient object detection. For the former, we apply transformer to a deterministic model, and explain that the effective structure modeling and global context modeling abilities lead to its superior performance compared with the CNN based frameworks. For the latter, we observe that both CNN and transformer based frameworks suffer greatly from the over-confidence issue, where the models tend to generate wrong predictions with high confidence. To estimate the reliability degree of both CNN- and transformer-based frameworks, we further present a latent variable model, namely inferential generative adversarial network (iGAN), based on the generative adversarial network (GAN). The stochastic attribute of the latent variable makes it convenient to estimate the predictive uncertainty, serving as an auxiliary output to evaluate the reliability of model prediction. Different from the conventional GAN, which defines the distribution of the latent variable as fixed standard normal distribution N(0,I), the proposed iGAN infers the latent variable by gradient-based Markov Chain Monte Carlo (MCMC), namely Langevin dynamics, leading to an input-dependent latent variable model. We apply our proposed iGAN to both fully and weakly supervised salient object detection, and explain that iGAN within the transformer framework leads to both accurate and reliable salient object detection.