# SeATrans: Learning Segmentation-Assisted diagnosis model via Transformer

arXiv: https://arxiv.org/abs/2206.05763 · HTML: https://mlsanity.com/paper/2206.05763/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 38 references that are in the map.
- First posted on arXiv 12 Jun 2022.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases](https://mlsanity.com/paper/2103.10697/) (2021)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1605.01397/) (2016)
- [Medical Image Analysis using Convolutional Neural Networks: A Review](https://mlsanity.com/paper/1709.02250/) (2017)
- [Attention Based Glaucoma Detection: A Large-scale Database and CNN Model](https://mlsanity.com/paper/1903.10831/) (2019)
- [Joint Optic Disc and Cup Segmentation Based on Multi-label Deep Network and Polar Transformation](https://mlsanity.com/paper/1801.00926/) (2018)
- [GAMMA Challenge:Glaucoma grAding from Multi-Modality imAges](https://mlsanity.com/paper/2202.06511/) (2022)
- [Disc-aware Ensemble Network for Glaucoma Screening from Fundus Image](https://mlsanity.com/paper/1805.07549/) (2018)
- [Leveraging Undiagnosed Data for Glaucoma Classification with Teacher-Student Learning](https://mlsanity.com/paper/2007.11355/) (2020)
- [Two-stage framework for optic disc localization and glaucoma classification in retinal fundus images using deep learning](https://mlsanity.com/paper/2005.14284/) (2020)
- [Robust Collaborative Learning of Patch-level and Image-level Annotations for Diabetic Retinopathy Grading from Fundus Image](https://mlsanity.com/paper/2008.00610/) (2020)
- [MAG-Net: Multi-task attention guided network for brain tumor segmentation and classification](https://mlsanity.com/paper/2107.12321/) (2021)

## Built on by (most cited first, 8 of 8)

- [MedSegDiff-V2: Diffusion based Medical Image Segmentation with Transformer](https://mlsanity.com/paper/2301.11798/) (2023)
- [Rethinking Boundary Detection in Deep Learning Models for Medical Image Segmentation](https://mlsanity.com/paper/2305.00678/) (2023)
- [Calibrate the inter-observer segmentation uncertainty via diagnosis-first principle](https://mlsanity.com/paper/2208.03016/) (2022)
- [Rethinking Boundary Detection in Deep Learning-Based Medical Image Segmentation](https://mlsanity.com/paper/2505.04652/) (2025)
- [An Efficient Person Clustering Algorithm for Open Checkout-free Groceries](https://mlsanity.com/paper/2208.02973/) (2022)
- [ExpNet: A unified network for Expert-Level Classification](https://mlsanity.com/paper/2211.15672/) (2022)
- [Not just Birds and Cars: Generic, Scalable and Explainable Models for Professional Visual Recognition](https://mlsanity.com/paper/2403.05703/) (2024)
- [MedFlowSeg: Flow Matching for Medical Image Segmentation with Frequency-Aware Attention](https://mlsanity.com/paper/2604.19675/) (2026)

## Abstract (arXiv)

Clinically, the accurate annotation of lesions/tissues can significantly facilitate the disease diagnosis. For example, the segmentation of optic disc/cup (OD/OC) on fundus image would facilitate the glaucoma diagnosis, the segmentation of skin lesions on dermoscopic images is helpful to the melanoma diagnosis, etc. With the advancement of deep learning techniques, a wide range of methods proved the lesions/tissues segmentation can also facilitate the automated disease diagnosis models. However, existing methods are limited in the sense that they can only capture static regional correlations in the images. Inspired by the global and dynamic nature of Vision Transformer, in this paper, we propose Segmentation-Assisted diagnosis Transformer (SeATrans) to transfer the segmentation knowledge to the disease diagnosis network. Specifically, we first propose an asymmetric multi-scale interaction strategy to correlate each single low-level diagnosis feature with multi-scale segmentation features. Then, an effective strategy called SeA-block is adopted to vitalize diagnosis feature via correlated segmentation features. To model the segmentation-diagnosis interaction, SeA-block first embeds the diagnosis feature based on the segmentation information via the encoder, and then transfers the embedding back to the diagnosis feature space by a decoder. Experimental results demonstrate that SeATrans surpasses a wide range of state-of-the-art (SOTA) segmentation-assisted diagnosis methods on several disease diagnosis tasks.