# Selecting the Best Optimizers for Deep Learning based Medical Image Segmentation

arXiv: https://arxiv.org/abs/2302.02289 · HTML: https://mlsanity.com/paper/2302.02289/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 53 references that are in the map.
- First posted on arXiv 5 Feb 2023.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [The Marginal Value of Adaptive Gradient Methods in Machine Learning](https://mlsanity.com/paper/1705.08292/) (2017)
- [MultiResUNet : Rethinking the U-Net Architecture for Multimodal Biomedical Image Segmentation](https://mlsanity.com/paper/1902.04049/) (2019)
- [Medical Image Segmentation Review: The success of U-Net](https://mlsanity.com/paper/2211.14830/) (2022)
- [A Novel Focal Tversky loss function with improved Attention U-Net for lesion segmentation](https://mlsanity.com/paper/1810.07842/) (2018)
- [The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation](https://mlsanity.com/paper/1611.09326/) (2016)
- [Anatomically Constrained Neural Networks (ACNN): Application to Cardiac Image Enhancement and Segmentation](https://mlsanity.com/paper/1705.08302/) (2017)
- [Evaluation of Algorithms for Multi-Modality Whole Heart Segmentation: An Open-Access Grand Challenge](https://mlsanity.com/paper/1902.07880/) (2019)
- [Capsules for Biomedical Image Segmentation](https://mlsanity.com/paper/2004.04736/) (2020)
- [YellowFin and the Art of Momentum Tuning](https://mlsanity.com/paper/1706.03471/) (2017)
- [Dynamic Linear Transformer for 3D Biomedical Image Segmentation](https://mlsanity.com/paper/2206.00771/) (2022)
- [The International Workshop on Osteoarthritis Imaging Knee MRI Segmentation Challenge: A Multi-Institute Evaluation and Analysis Framework on a Standardized Dataset](https://mlsanity.com/paper/2004.14003/) (2020)
- [Deep Learning in Cardiology](https://mlsanity.com/paper/1902.11122/) (2019)
- [CardiacNET: Segmentation of Left Atrium and Proximal Pulmonary Veins from MRI Using Multi-View CNN](https://mlsanity.com/paper/1705.06333/) (2017)
- [Monte-Carlo Sampling applied to Multiple Instance Learning for Histological Image Classification](https://mlsanity.com/paper/1812.11560/) (2018)
- [DilatedSegNet: A Deep Dilated Segmentation Network for Polyp Segmentation](https://mlsanity.com/paper/2210.13595/) (2022)
- [Transformer based Generative Adversarial Network for Liver Segmentation](https://mlsanity.com/paper/2205.10663/) (2022)
- [An Efficient Multi-Scale Fusion Network for 3D Organ at Risk (OAR) Segmentation](https://mlsanity.com/paper/2208.07417/) (2022)
- [Weakly Supervised Segmentation by A Deep Geodesic Prior](https://mlsanity.com/paper/1908.06498/) (2019)
- [Multi-Scale Fusion Methodologies for Head and Neck Tumor Segmentation](https://mlsanity.com/paper/2210.16704/) (2022)

## Built on by (most cited first, 1 of 1)

- [Uncertainty-Guided Cross Attention Ensemble Mean Teacher for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2412.15380/) (2024)

## Abstract (arXiv)

The goal of this work is to identify the best optimizers for deep learning in the context of cardiac image segmentation and to provide guidance on how to design segmentation networks with effective optimization strategies. Adaptive learning helps with fast convergence by starting with a larger learning rate (LR) and gradually decreasing it. Momentum optimizers are particularly effective at quickly optimizing neural networks within the accelerated schemes category. By revealing the potential interplay between these two types of algorithms (LR and momentum optimizers or momentum rate (MR) in short), in this article, we explore the two variants of SGD algorithms in a single setting. We suggest using cyclic learning as the base optimizer and integrating optimal values of learning rate and momentum rate. We investigated the relationship of LR and MR under an important problem of medical image segmentation of cardiac structures from MRI and CT scans. We conducted experiments using the cardiac imaging dataset from the ACDC challenge of MICCAI 2017, and four different architectures shown to be successful for cardiac image segmentation problems. Our comprehensive evaluations demonstrated that the proposed optimizer achieved better results (over a 2% improvement in the dice metric) than other optimizers in deep learning literature with similar or lower computational cost in both single and multi-object segmentation settings. We hypothesized that combination of accelerated and adaptive optimization methods can have a drastic effect in medical image segmentation performances. To this end, we proposed a new cyclic optimization method (textitCLMR) to address the efficiency and accuracy problems in deep learning based medical image segmentation. The proposed strategy yielded better generalization in comparison to adaptive optimizers.