# Histogram of Oriented Gradients Meet Deep Learning: A Novel Multi-task Deep Network for Medical Image Semantic Segmentation

arXiv: https://arxiv.org/abs/2204.01712 · HTML: https://mlsanity.com/paper/2204.01712/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 60 references that are in the map.
- First posted on arXiv 2 Apr 2022.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection](https://mlsanity.com/paper/2005.09007/) (2020)
- [Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation](https://mlsanity.com/paper/1603.05959/) (2016)
- [LinkNet: Exploiting Encoder Representations for Efficient Semantic Segmentation](https://mlsanity.com/paper/1707.03718/) (2017)
- [2017 Robotic Instrument Segmentation Challenge](https://mlsanity.com/paper/1902.06426/) (2019)
- [Gated-SCNN: Gated Shape CNNs for Semantic Segmentation](https://mlsanity.com/paper/1907.05740/) (2019)
- [Medical Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2009.13120/) (2020)
- [Pyramid Attention Network for Semantic Segmentation](https://mlsanity.com/paper/1805.10180/) (2018)
- [Medical Image Analysis using Convolutional Neural Networks: A Review](https://mlsanity.com/paper/1709.02250/) (2017)
- [Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning](https://mlsanity.com/paper/1803.01207/) (2018)
- [TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation](https://mlsanity.com/paper/1801.05746/) (2018)
- [Learning Representations by Predicting Bags of Visual Words](https://mlsanity.com/paper/2002.12247/) (2020)
- [Semi-supervised few-shot learning for medical image segmentation](https://mlsanity.com/paper/2003.08462/) (2020)
- [Self-Paced Contrastive Learning for Semi-supervised Medical Image Segmentation with Meta-labels](https://mlsanity.com/paper/2107.13741/) (2021)
- [Synthetic and Real Inputs for Tool Segmentation in Robotic Surgery](https://mlsanity.com/paper/2007.09107/) (2020)
- [Deep Residual Learning for Instrument Segmentation in Robotic Surgery](https://mlsanity.com/paper/1703.08580/) (2017)
- [Spatial Aggregation of Holistically-Nested Convolutional Neural Networks for Automated Pancreas Localization and Segmentation](https://mlsanity.com/paper/1702.00045/) (2017)
- [Simultaneous Segmentation and Classification of Bone Surfaces from Ultrasound Using a Multi-feature Guided CNN](https://mlsanity.com/paper/1806.09766/) (2018)
- [A Deep Learning based Joint Segmentation and Classification Framework for Glaucoma Assesment in Retinal Color Fundus Images](https://mlsanity.com/paper/1808.01355/) (2018)

## Built on by (most cited first, 4 of 4)

- [SimuScope: Realistic Endoscopic Synthetic Dataset Generation through Surgical Simulation and Diffusion Models](https://mlsanity.com/paper/2412.02332/) (2024)
- [Multitask Learning in Minimally Invasive Surgical Vision: A Review](https://mlsanity.com/paper/2401.08256/) (2024)
- [A Dual-Task Synergy-Driven Generalization Framework for Pancreatic Cancer Segmentation in CT Scans](https://mlsanity.com/paper/2505.01644/) (2025)
- [A Comparison of Selected Image Transformation Techniques for Malware Classification](https://mlsanity.com/paper/2509.10838/) (2025)

## Abstract (arXiv)

We present our novel deep multi-task learning method for medical image segmentation. Existing multi-task methods demand ground truth annotations for both the primary and auxiliary tasks. Contrary to it, we propose to generate the pseudo-labels of an auxiliary task in an unsupervised manner. To generate the pseudo-labels, we leverage Histogram of Oriented Gradients (HOGs), one of the most widely used and powerful hand-crafted features for detection. Together with the ground truth semantic segmentation masks for the primary task and pseudo-labels for the auxiliary task, we learn the parameters of the deep network to minimise the loss of both the primary task and the auxiliary task jointly. We employed our method on two powerful and widely used semantic segmentation networks: UNet and U2Net to train in a multi-task setup. To validate our hypothesis, we performed experiments on two different medical image segmentation data sets. From the extensive quantitative and qualitative results, we observe that our method consistently improves the performance compared to the counter-part method. Moreover, our method is the winner of FetReg Endovis Sub-challenge on Semantic Segmentation organised in conjunction with MICCAI 2021.