# SynergyNet: Bridging the Gap between Discrete and Continuous Representations for Precise Medical Image Segmentation

arXiv: https://arxiv.org/abs/2310.17764 · HTML: https://mlsanity.com/paper/2310.17764/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 28 references that are in the map.
- First posted on arXiv 26 Oct 2023.

## Builds on (most cited first, 21 of 21)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1710.05006/) (2017)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentation](https://mlsanity.com/paper/2207.08518/) (2022)
- [Bi-Directional ConvLSTM U-Net with Densley Connected Convolutions](https://mlsanity.com/paper/1909.00166/) (2019)
- [Rethinking Minimal Sufficient Representation in Contrastive Learning](https://mlsanity.com/paper/2203.07004/) (2022)
- [MALUNet: A Multi-Attention and Light-weight UNet for Skin Lesion Segmentation](https://mlsanity.com/paper/2211.01784/) (2022)
- [Y-Net: A Spatiospectral Dual-Encoder Networkfor Medical Image Segmentation](https://mlsanity.com/paper/2204.07613/) (2022)
- [Transformer-based out-of-distribution detection for clinically safe segmentation](https://mlsanity.com/paper/2205.10650/) (2022)
- [Discrete neural representations for explainable anomaly detection](https://mlsanity.com/paper/2112.05585/) (2021)
- [Vector Quantisation for Robust Segmentation](https://mlsanity.com/paper/2207.01919/) (2022)

## Built on by (most cited first, 9 of 9)

- [Large Scale MRI Collection and Segmentation of Cirrhotic Liver](https://mlsanity.com/paper/2410.16296/) (2024)
- [Adaptive Aggregation Weights for Federated Segmentation of Pancreas MRI](https://mlsanity.com/paper/2410.22530/) (2024)
- [OTCXR: Rethinking Self-supervised Alignment using Optimal Transport for Chest X-ray Analysis](https://mlsanity.com/paper/2404.11868/) (2024)
- [Rethinking Intermediate Layers design in Knowledge Distillation for Kidney and Liver Tumor Segmentation](https://mlsanity.com/paper/2311.16700/) (2023)
- [Harmonized Spatial and Spectral Learning for Robust and Generalized Medical Image Segmentation](https://mlsanity.com/paper/2401.10373/) (2024)
- [CoBooM: Codebook Guided Bootstrapping for Medical Image Representation Learning](https://mlsanity.com/paper/2408.04262/) (2024)
- [Towards Synergistic Deep Learning Models for Volumetric Cirrhotic Liver Segmentation in MRIs](https://mlsanity.com/paper/2408.04491/) (2024)
- [SegDT: A Diffusion Transformer-Based Segmentation Model for Medical Imaging](https://mlsanity.com/paper/2507.15595/) (2025)
- [RobSurv: Vector Quantization-Based Multi-Modal Learning for Robust Cancer Survival Prediction](https://mlsanity.com/paper/2505.02529/) (2025)

## Abstract (arXiv)

In recent years, continuous latent space (CLS) and discrete latent space (DLS) deep learning models have been proposed for medical image analysis for improved performance. However, these models encounter distinct challenges. CLS models capture intricate details but often lack interpretability in terms of structural representation and robustness due to their emphasis on low-level features. Conversely, DLS models offer interpretability, robustness, and the ability to capture coarse-grained information thanks to their structured latent space. However, DLS models have limited efficacy in capturing fine-grained details. To address the limitations of both DLS and CLS models, we propose SynergyNet, a novel bottleneck architecture designed to enhance existing encoder-decoder segmentation frameworks. SynergyNet seamlessly integrates discrete and continuous representations to harness complementary information and successfully preserves both fine and coarse-grained details in the learned representations. Our extensive experiment on multi-organ segmentation and cardiac datasets demonstrates that SynergyNet outperforms other state of the art methods, including TransUNet: dice scores improving by 2.16%, and Hausdorff scores improving by 11.13%, respectively. When evaluating skin lesion and brain tumor segmentation datasets, we observe a remarkable improvement of 1.71% in Intersection-over Union scores for skin lesion segmentation and of 8.58% for brain tumor segmentation. Our innovative approach paves the way for enhancing the overall performance and capabilities of deep learning models in the critical domain of medical image analysis.