# JOINEDTrans: Prior Guided Multi-task Transformer for Joint Optic Disc/Cup Segmentation and Fovea Detection

arXiv: https://arxiv.org/abs/2305.11504 · HTML: https://mlsanity.com/paper/2305.11504/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 61 references that are in the map.
- First posted on arXiv 19 May 2023.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [REFUGE Challenge: A Unified Framework for Evaluating Automated Methods for Glaucoma Assessment from Fundus Photographs](https://mlsanity.com/paper/1910.03667/) (2019)
- [Topology-Preserving Deep Image Segmentation](https://mlsanity.com/paper/1906.05404/) (2019)
- [Joint Optic Disc and Cup Segmentation Based on Multi-label Deep Network and Polar Transformation](https://mlsanity.com/paper/1801.00926/) (2018)
- [Medical Image Segmentation Using Squeeze-and-Expansion Transformers](https://mlsanity.com/paper/2105.09511/) (2021)
- [GAMMA Challenge:Glaucoma grAding from Multi-Modality imAges](https://mlsanity.com/paper/2202.06511/) (2022)
- [Task Adaptive Parameter Sharing for Multi-Task Learning](https://mlsanity.com/paper/2203.16708/) (2022)
- [Unsupervised Learning of Landmarks by Descriptor Vector Exchange](https://mlsanity.com/paper/1908.06427/) (2019)
- [Towards a glaucoma risk index based on simulated hemodynamics from fundus images](https://mlsanity.com/paper/1805.10273/) (2018)
- [Deep Convolutional Neural Networks with Spatial Regularization, Volume and Star-shape Priori for Image Segmentation](https://mlsanity.com/paper/2002.03989/) (2020)
- [High-level Prior-based Loss Functions for Medical Image Segmentation: A Survey](https://mlsanity.com/paper/2011.08018/) (2020)
- [Controllable Dynamic Multi-Task Architectures](https://mlsanity.com/paper/2203.14949/) (2022)
- [AADG: Automatic Augmentation for Domain Generalization on Retinal Image Segmentation](https://mlsanity.com/paper/2207.13249/) (2022)
- [BSDA-Net: A Boundary Shape and Distance Aware Joint Learning Framework for Segmenting and Classifying OCTA Images](https://mlsanity.com/paper/2107.04823/) (2021)
- [Convex Shape Prior for Deep Neural Convolution Network based Eye Fundus Images Segmentation](https://mlsanity.com/paper/2005.07476/) (2020)
- [COROLLA: An Efficient Multi-Modality Fusion Framework with Supervised Contrastive Learning for Glaucoma Grading](https://mlsanity.com/paper/2201.03795/) (2022)
- [Transfer Learning based Detection of Diabetic Retinopathy from Small Dataset](https://mlsanity.com/paper/1905.07203/) (2019)
- [MT-TransUNet: Mediating Multi-Task Tokens in Transformers for Skin Lesion Segmentation and Classification](https://mlsanity.com/paper/2112.01767/) (2021)
- [Segmentation of optic disc, fovea and retinal vasculature using a single convolutional neural network](https://mlsanity.com/paper/1702.00509/) (2017)
- [Bilateral-ViT for Robust Fovea Localization](https://mlsanity.com/paper/2110.09860/) (2021)
- [Convex Shape Representation with Binary Labels for Image Segmentation: Models and Fast Algorithms](https://mlsanity.com/paper/2002.09600/) (2020)
- [U-Net with Hierarchical Bottleneck Attention for Landmark Detection in Fundus Images of the Degenerated Retina](https://mlsanity.com/paper/2107.04721/) (2021)

## Abstract (arXiv)

Deep learning-based image segmentation and detection models have largely improved the efficiency of analyzing retinal landmarks such as optic disc (OD), optic cup (OC), and fovea. However, factors including ophthalmic disease-related lesions and low image quality issues may severely complicate automatic OD/OC segmentation and fovea detection. Most existing works treat the identification of each landmark as a single task, and take into account no prior information. To address these issues, we propose a prior guided multi-task transformer framework for joint OD/OC segmentation and fovea detection, named JOINEDTrans. JOINEDTrans effectively combines various spatial features of the fundus images, relieving the structural distortions induced by lesions and other imaging issues. It contains a segmentation branch and a detection branch. To be noted, we employ an encoder pretrained in a vessel segmentation task to effectively exploit the positional relationship among vessel, OD/OC, and fovea, successfully incorporating spatial prior into the proposed JOINEDTrans framework. There are a coarse stage and a fine stage in JOINEDTrans. In the coarse stage, OD/OC coarse segmentation and fovea heatmap localization are obtained through a joint segmentation and detection module. In the fine stage, we crop regions of interest for subsequent refinement and use predictions obtained in the coarse stage to provide additional information for better performance and faster convergence. Experimental results demonstrate that JOINEDTrans outperforms existing state-of-the-art methods on the publicly available GAMMA, REFUGE, and PALM fundus image datasets. We make our code available at https://github.com/HuaqingHe/JOINEDTrans