# FoPro-KD: Fourier Prompted Effective Knowledge Distillation for Long-Tailed Medical Image Recognition

arXiv: https://arxiv.org/abs/2305.17421 · HTML: https://mlsanity.com/paper/2305.17421/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Distillation for medical imaging, one of 12 directions in Knowledge distillation.
- Builds on 34 of its 44 references that are in the map.
- First posted on arXiv 27 May 2023.

## Placed in

- Knowledge distillation / [Distillation for medical imaging](https://mlsanity.com/distill/distillation-for-medical-imaging/)

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [Deep Long-Tailed Learning: A Survey](https://mlsanity.com/paper/2110.04596/) (2021)
- [Balanced Meta-Softmax for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2007.10740/) (2020)
- [BCN20000: Dermoscopic Lesions in the Wild](https://mlsanity.com/paper/1908.02288/) (2019)
- [Knowledge Distillation with the Reused Teacher Classifier](https://mlsanity.com/paper/2203.14001/) (2022)
- [Improving Vision Transformers by Revisiting High-frequency Components](https://mlsanity.com/paper/2204.00993/) (2022)
- [LPT: Long-tailed Prompt Tuning for Image Classification](https://mlsanity.com/paper/2210.01033/) (2022)
- [Contrastive Model Inversion for Data-Free Knowledge Distillation](https://mlsanity.com/paper/2105.08584/) (2021)
- [Label-Free Liver Tumor Segmentation](https://mlsanity.com/paper/2303.14869/) (2023)
- [RDA: Robust Domain Adaptation via Fourier Adversarial Attacking](https://mlsanity.com/paper/2106.02874/) (2021)
- [Balanced-MixUp for Highly Imbalanced Medical Image Classification](https://mlsanity.com/paper/2109.09850/) (2021)
- [Long-Tailed Classification of Thorax Diseases on Chest X-Ray: A New Benchmark Study](https://mlsanity.com/paper/2208.13365/) (2022)
- [Invariant Feature Learning for Generalized Long-Tailed Classification](https://mlsanity.com/paper/2207.09504/) (2022)
- [Data-Free Knowledge Amalgamation via Group-Stack Dual-GAN](https://mlsanity.com/paper/2003.09088/) (2020)
- [MaxStyle: Adversarial Style Composition for Robust Medical Image Segmentation](https://mlsanity.com/paper/2206.01737/) (2022)
- [Balanced Knowledge Distillation for Long-tailed Learning](https://mlsanity.com/paper/2104.10510/) (2021)
- [FVP: Fourier Visual Prompting for Source-Free Unsupervised Domain Adaptation of Medical Image Segmentation](https://mlsanity.com/paper/2304.13672/) (2023)
- [ProSFDA: Prompt Learning based Source-free Domain Adaptation for Medical Image Segmentation](https://mlsanity.com/paper/2211.11514/) (2022)
- [Differences between human and machine perception in medical diagnosis](https://mlsanity.com/paper/2011.14036/) (2020)
- [Well-classified Examples are Underestimated in Classification with Deep Neural Networks](https://mlsanity.com/paper/2110.06537/) (2021)

## Built on by (most cited first, 10 of 10)

- [Self-Prompting Large Vision Models for Few-Shot Medical Image Segmentation](https://mlsanity.com/paper/2308.07624/) (2023)
- [Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlooks](https://mlsanity.com/paper/2410.02331/) (2024)
- [Towards Distribution-Agnostic Generalized Category Discovery](https://mlsanity.com/paper/2310.01376/) (2023)
- [Robustly Optimized Deep Feature Decoupling Network for Fatty Liver Diseases Detection](https://mlsanity.com/paper/2406.17338/) (2024)
- [MultiEYE: Dataset and Benchmark for OCT-Enhanced Retinal Disease Recognition from Fundus Images](https://mlsanity.com/paper/2412.09402/) (2024)
- [Federated Model Aggregation via Self-Supervised Priors for Highly Imbalanced Medical Image Classification](https://mlsanity.com/paper/2307.14959/) (2023)
- [Intelligence as Managed Autonomy: Failure, Escalation, and Governance for Agentic AI Systems](https://mlsanity.com/paper/2605.27628/) (2026)
- [Enhancing Single-Slice Segmentation with 3D-to-2D Unpaired Scan Distillation](https://mlsanity.com/paper/2406.12254/) (2024)
- [Multiple Teachers-Meticulous Student: A Domain Adaptive Meta-Knowledge Distillation Model for Medical Image Classification](https://mlsanity.com/paper/2403.11226/) (2024)
- [Dynamic Weight Adjustment for Knowledge Distillation: Leveraging Vision Transformer for High-Accuracy Lung Cancer Detection and Real-Time Deployment](https://mlsanity.com/paper/2510.20438/) (2025)

## Abstract (arXiv)

Representational transfer from publicly available models is a promising technique for improving medical image classification, especially in long-tailed datasets with rare diseases. However, existing methods often overlook the frequency-dependent behavior of these models, thereby limiting their effectiveness in transferring representations and generalizations to rare diseases. In this paper, we propose FoPro-KD, a novel framework that leverages the power of frequency patterns learned from frozen pre-trained models to enhance their transferability and compression, presenting a few unique insights: 1) We demonstrate that leveraging representations from publicly available pre-trained models can substantially improve performance, specifically for rare classes, even when utilizing representations from a smaller pre-trained model. 2) We observe that pre-trained models exhibit frequency preferences, which we explore using our proposed Fourier Prompt Generator (FPG), allowing us to manipulate specific frequencies in the input image, enhancing the discriminative representational transfer. 3) By amplifying or diminishing these frequencies in the input image, we enable Effective Knowledge Distillation (EKD). EKD facilitates the transfer of knowledge from pre-trained models to smaller models. Through extensive experiments in long-tailed gastrointestinal image recognition and skin lesion classification, where rare diseases are prevalent, our FoPro-KD framework outperforms existing methods, enabling more accessible medical models for rare disease classification. Code is available at https://github.com/xmed-lab/FoPro-KD.