# HiFuse: Hierarchical Multi-Scale Feature Fusion Network for Medical Image Classification

arXiv: https://arxiv.org/abs/2209.10218 · HTML: https://mlsanity.com/paper/2209.10218/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 64 references that are in the map.
- First posted on arXiv 21 Sep 2022.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014)
- [Visual Transformers: Token-based Image Representation and Processing for Computer Vision](https://mlsanity.com/paper/2006.03677/) (2020)
- [Medical Transformer: Gated Axial-Attention for Medical Image Segmentation](https://mlsanity.com/paper/2102.10662/) (2021)
- [CMT: Convolutional Neural Networks Meet Vision Transformers](https://mlsanity.com/paper/2107.06263/) (2021)
- [TransBTS: Multimodal Brain Tumor Segmentation Using Transformer](https://mlsanity.com/paper/2103.04430/) (2021)
- [TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation](https://mlsanity.com/paper/2102.08005/) (2021)
- [Attention Mechanisms in Computer Vision: A Survey](https://mlsanity.com/paper/2111.07624/) (2021)
- [A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises](https://mlsanity.com/paper/2008.09104/) (2020)

## Built on by (most cited first, 30 of 30)

- [CardiacNet: Learning to Reconstruct Abnormalities for Cardiac Disease Assessment from Echocardiogram Videos](https://mlsanity.com/paper/2410.20769/) (2024)
- [Towards a Medical AI Scientist](https://mlsanity.com/paper/2603.28589/) (2026)
- [Medical Image Classification with KAN-Integrated Transformers and Dilated Neighborhood Attention](https://mlsanity.com/paper/2502.13693/) (2025)
- [Federated Distillation for Medical Image Classification: Towards Trustworthy Computer-Aided Diagnosis](https://mlsanity.com/paper/2407.02261/) (2024)
- [FMDNN: A Fuzzy-guided Multi-granular Deep Neural Network for Histopathological Image Classification](https://mlsanity.com/paper/2407.15312/) (2024)
- [MMLNB: Multi-Modal Learning for Neuroblastoma Subtyping Classification Assisted with Textual Description Generation](https://mlsanity.com/paper/2503.12927/) (2025)
- [MedFormer: Hierarchical Medical Vision Transformer with Content-Aware Dual Sparse Selection Attention](https://mlsanity.com/paper/2507.02488/) (2025)
- [FMRT: Learning Accurate Feature Matching with Reconciliatory Transformer](https://mlsanity.com/paper/2310.13605/) (2023)
- [MIAFEx: An Attention-based Feature Extraction Method for Medical Image Classification](https://mlsanity.com/paper/2501.08562/) (2025)
- [DCL-SE: Dynamic Curriculum Learning for Spatiotemporal Encoding of Brain Imaging](https://mlsanity.com/paper/2511.15151/) (2025)
- [MATCNN: Infrared and Visible Image Fusion Method Based on Multi-scale CNN with Attention Transformer](https://mlsanity.com/paper/2502.01959/) (2025)
- [WaRA: Wavelet Low-Rank Adaptation for Medical Image Classification](https://mlsanity.com/paper/2506.24092/) (2025)
- [SAFE Quantum Machine Learning with Variational Quantum Classifiers](https://mlsanity.com/paper/2605.16067/) (2026)
- [EAFP-Med: An Efficient Adaptive Feature Processing Module Based on Prompts for Medical Image Detection](https://mlsanity.com/paper/2311.15540/) (2023)
- [DFE-IANet: A Method for Polyp Image Classification Based on Dual-domain Feature Extraction and Interaction Attention](https://mlsanity.com/paper/2407.20843/) (2024)
- [Enhanced Cascade Prostate Cancer Classifier in mp-MRI Utilizing Recall Feedback Adaptive Loss and Prior Knowledge-Based Feature Extraction](https://mlsanity.com/paper/2408.09746/) (2024)
- [MpoxMamba: A Grouped Mamba-based Lightweight Hybrid Network for Mpox Detection](https://mlsanity.com/paper/2409.04218/) (2024)
- [Leveraging Vision-Language Embeddings for Zero-Shot Learning in Histopathology Images](https://mlsanity.com/paper/2503.10731/) (2025)
- [Excavate the potential of Single-Scale Features: A Decomposition Network for Water-Related Optical Image Enhancement](https://mlsanity.com/paper/2508.04123/) (2025)
- [HiPerformer: A High-Performance Global-Local Segmentation Model with Modular Hierarchical Fusion Strategy](https://mlsanity.com/paper/2509.20280/) (2025)
- [ITC-RWKV: Interactive Tissue-Cell Modeling with Recurrent Key-Value Aggregation for Histopathological Subtyping](https://mlsanity.com/paper/2510.21479/) (2025)
- [hYOLO Model: Enhancing Object Classification with Hierarchical Context in YOLOv8](https://mlsanity.com/paper/2510.23278/) (2025)
- [A Hybrid Framework Bridging CNN and ViT based on Theory of Evidence for Diabetic Retinopathy Grading](https://mlsanity.com/paper/2510.26315/) (2025)
- [SAMora: Enhancing SAM through Hierarchical Self-Supervised Pre-Training for Medical Images](https://mlsanity.com/paper/2511.08626/) (2025)
- [AGGRNet: Selective Feature Extraction and Aggregation for Enhanced Medical Image Classification](https://mlsanity.com/paper/2511.12382/) (2025)
- [Cross-modal ultra-scale learning with tri-modalities of renal biopsy images for glomerular multi-disease auxiliary diagnosis](https://mlsanity.com/paper/2512.15171/) (2025)
- [Entropy-Guided Self-Supervised Learning for Medical Image Classification](https://mlsanity.com/paper/2605.21970/) (2026)
- [Recurrent Contrastive Learning for Imbalanced Medical Image Classification](https://mlsanity.com/paper/2608.03304/) (2026)
- [Integrating Local Detail and Global Context: A Dual-Input Multi-Task Learning Framework for Bone Tumor Diagnosis](https://mlsanity.com/paper/2609.28732/) (2026)
- [CFCH: Coarse-Fine Collaborative Hierarchical Learning for Anterior Segment Disease Analysis](https://mlsanity.com/paper/2609.32559/) (2026)

## Abstract (arXiv)

Medical image classification has developed rapidly under the impetus of the convolutional neural network (CNN). Due to the fixed size of the receptive field of the convolution kernel, it is difficult to capture the global features of medical images. Although the self-attention-based Transformer can model long-range dependencies, it has high computational complexity and lacks local inductive bias. Much research has demonstrated that global and local features are crucial for image classification. However, medical images have a lot of noisy, scattered features, intra-class variation, and inter-class similarities. This paper proposes a three-branch hierarchical multi-scale feature fusion network structure termed as HiFuse for medical image classification as a new method. It can fuse the advantages of Transformer and CNN from multi-scale hierarchies without destroying the respective modeling so as to improve the classification accuracy of various medical images. A parallel hierarchy of local and global feature blocks is designed to efficiently extract local features and global representations at various semantic scales, with the flexibility to model at different scales and linear computational complexity relevant to image size. Moreover, an adaptive hierarchical feature fusion block (HFF block) is designed to utilize the features obtained at different hierarchical levels comprehensively. The HFF block contains spatial attention, channel attention, residual inverted MLP, and shortcut to adaptively fuse semantic information between various scale features of each branch. The accuracy of our proposed model on the ISIC2018 dataset is 7.6% higher than baseline, 21.5% on the Covid-19 dataset, and 10.4% on the Kvasir dataset. Compared with other advanced models, the HiFuse model performs the best. Our code is open-source and available from https://github.com/huoxiangzuo/HiFuse.