# MIFR: A Modality-Invariant and Fair Representation Framework for Skin Disease Classification

arXiv: https://arxiv.org/abs/2608.18774 · HTML: https://mlsanity.com/paper/2608.18774/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 19 references that are in the map.
- First posted on arXiv 19 Aug 2026.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [BCN20000: Dermoscopic Lesions in the Wild](https://mlsanity.com/paper/1908.02288/) (2019)
- [Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset](https://mlsanity.com/paper/2104.09957/) (2021)
- [Disparities in Dermatology AI: Assessments Using Diverse Clinical Images](https://mlsanity.com/paper/2111.08006/) (2021)
- [PAD-UFES-20: a skin lesion dataset composed of patient data and clinical images collected from smartphones](https://mlsanity.com/paper/2007.00478/) (2020)
- [A Multimodal Vision Foundation Model for Clinical Dermatology](https://mlsanity.com/paper/2410.15038/) (2024)
- [FairDisCo: Fairer AI in Dermatology via Disentanglement Contrastive Learning](https://mlsanity.com/paper/2208.10013/) (2022)
- [FairAdaBN: Mitigating unfairness with adaptive batch normalization and its application to dermatological disease classification](https://mlsanity.com/paper/2303.08325/) (2023)
- [Achieving Fairness Through Channel Pruning for Dermatological Disease Diagnosis](https://mlsanity.com/paper/2405.08681/) (2024)
- [PatchAlign:Fair and Accurate Skin Disease Image Classification by Alignment with Clinical Labels](https://mlsanity.com/paper/2409.04975/) (2024)
- [eSkinHealth: A Multimodal Dataset for Neglected Tropical Skin Diseases](https://mlsanity.com/paper/2508.18608/) (2025)

## Abstract (arXiv)

Skin diseases represent a major global public health burden, yet machine learning tools developed to assist in their diagnosis suffer from two critical limitations: reliance on only one modality for diagnosis and systematic performance disparities across skin tones. While existing approaches address each challenge separately, this work proposes a modality-invariant framework with fair representation (MIFR) for skin disease classification. The architecture pairs clinical photographs with dermoscopic images using ViT-based encoders, projecting each input into a high-dimensional embedding space via modality-specific projection heads. The resulting model is trained with a five-component multi-objective loss including weighted cross-entropy for classification, confusion and skin-type classification losses for fairness, per-modality supervised contrastive loss for class alignment, and a modality-invariance loss for clinical and dermoscopic modality alignment. Experiments on the HIBA+Derm7pt paired dataset and the external PAD-UFES-20 and ISIC 2019 datasets showed that modality-invariant representation learning provides competitive predictive performance compare to relevant baseline models and competitive fairness on the internal dataset. t-SNE visualizations confirmed that clinical and dermoscopic embeddings of the same disease are geometrically aligned, validating the joint objectives.