# FairTune: Optimizing Parameter Efficient Fine Tuning for Fairness in Medical Image Analysis

arXiv: https://arxiv.org/abs/2310.05055 · HTML: https://mlsanity.com/paper/2310.05055/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 90 references that are in the map.
- First posted on arXiv 8 Oct 2023.

## Builds on (most cited first, 30 of 55)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [A Reductions Approach to Fair Classification](https://mlsanity.com/paper/1803.02453/) (2018)
- [Do Better ImageNet Models Transfer Better?](https://mlsanity.com/paper/1805.08974/) (2018)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [SWAD: Domain Generalization by Seeking Flat Minima](https://mlsanity.com/paper/2102.08604/) (2021)
- [Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis](https://mlsanity.com/paper/2111.14791/) (2021)
- [Fairness in Machine Learning: A Survey](https://mlsanity.com/paper/2010.04053/) (2020)
- [Learning Adversarially Fair and Transferable Representations](https://mlsanity.com/paper/1802.06309/) (2018)
- [Adversarial Feature Learning](https://mlsanity.com/paper/1605.09782/) (2016)
- [Big Self-Supervised Models Advance Medical Image Classification](https://mlsanity.com/paper/2101.05224/) (2021)
- [Customized Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.13785/) (2023)
- [Learning Not to Learn: Training Deep Neural Networks with Biased Data](https://mlsanity.com/paper/1812.10352/) (2018)
- [Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation](https://mlsanity.com/paper/1911.11834/) (2019)
- [Empirical Risk Minimization under Fairness Constraints](https://mlsanity.com/paper/1802.08626/) (2018)
- [Non-stochastic Best Arm Identification and Hyperparameter Optimization](https://mlsanity.com/paper/1502.07943/) (2015)
- [Adversarially Learned Inference](https://mlsanity.com/paper/1606.00704/) (2016)
- [A System for Massively Parallel Hyperparameter Tuning](https://mlsanity.com/paper/1810.05934/) (2018)
- [Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset](https://mlsanity.com/paper/2104.09957/) (2021)
- [Fair Regression: Quantitative Definitions and Reduction-based Algorithms](https://mlsanity.com/paper/1905.12843/) (2019)

## Built on by (most cited first, 23 of 23)

- [FairMedFM: Fairness Benchmarking for Medical Imaging Foundation Models](https://mlsanity.com/paper/2407.00983/) (2024)
- [Parameter Efficient Fine Tuning: A Comprehensive Analysis Across Applications](https://mlsanity.com/paper/2404.13506/) (2024)
- [OxonFair: A Flexible Toolkit for Algorithmic Fairness](https://mlsanity.com/paper/2407.13710/) (2024)
- [Rethinking Fair Representation Learning for Performance-Sensitive Tasks](https://mlsanity.com/paper/2410.04120/) (2024)
- [Embedded Visual Prompt Tuning](https://mlsanity.com/paper/2407.01003/) (2024)
- [BMFT: Achieving Fairness via Bias-based Weight Masking Fine-tuning](https://mlsanity.com/paper/2408.06890/) (2024)
- [CheXGenBench: A Unified Benchmark For Fidelity, Privacy and Utility of Synthetic Chest Radiographs](https://mlsanity.com/paper/2505.10496/) (2025)
- [MemControl: Mitigating Memorization in Diffusion Models via Automated Parameter Selection](https://mlsanity.com/paper/2405.19458/) (2024)
- [On Biases in a UK Biobank-based Retinal Image Classification Model](https://mlsanity.com/paper/2408.02676/) (2024)
- [Benchmarking Foundation Models and Parameter-Efficient Fine-Tuning for Prognosis Prediction in Medical Imaging](https://mlsanity.com/paper/2506.18434/) (2025)
- [FairLoRA: Unpacking Bias Mitigation in Vision Models with Fairness-Driven Low-Rank Adaptation](https://mlsanity.com/paper/2410.17358/) (2024)
- [On Fairness of Task Arithmetic: The Role of Task Vectors](https://mlsanity.com/paper/2505.24262/) (2025)
- [Capacity Control is an Effective Memorization Mitigation Mechanism in Text-Conditional Diffusion Models](https://mlsanity.com/paper/2410.22149/) (2024)
- [A Multi-Objective Evaluation Framework for Analyzing Utility-Fairness Trade-Offs in Machine Learning Systems](https://mlsanity.com/paper/2503.11120/) (2025)
- [OxEnsemble: Fair Ensembles for Low-Data Classification](https://mlsanity.com/paper/2512.09665/) (2025)
- [Stride-Net: Fairness-Aware Disentangled Representation Learning for Chest X-Ray Diagnosis](https://mlsanity.com/paper/2602.10875/) (2026)
- [FairLLaVA: Fairness-Aware Parameter-Efficient Fine-Tuning for Large Vision-Language Assistants](https://mlsanity.com/paper/2603.26008/) (2026)
- [Fairness-Aware Low-Rank Representation Fine-Tuning](https://mlsanity.com/paper/2503.05684/) (2025)
- [SWiFT: Soft-Mask Weight Fine-tuning for Bias Mitigation](https://mlsanity.com/paper/2508.18826/) (2025)
- [Toward Robust Medical Fairness: Debiased Dual-Modal Alignment via Text-Guided Attribute-Disentangled Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2508.18886/) (2025)
- [Fairness-Aware Fine-Tuning of Vision-Language Models for Medical Glaucoma Diagnosis](https://mlsanity.com/paper/2512.03477/) (2025)
- [Self-Auditing Parameter-Efficient Fine-Tuning for Few-Shot 3D Medical Image Segmentation](https://mlsanity.com/paper/2603.05822/) (2026)
- [Subgroup performance analysis of adaptation strategies for chest X-ray foundation models](https://mlsanity.com/paper/2608.19078/) (2026)

## Abstract (arXiv)

Training models with robust group fairness properties is crucial in ethically sensitive application areas such as medical diagnosis. Despite the growing body of work aiming to minimise demographic bias in AI, this problem remains challenging. A key reason for this challenge is the fairness generalisation gap: High-capacity deep learning models can fit all training data nearly perfectly, and thus also exhibit perfect fairness during training. In this case, bias emerges only during testing when generalisation performance differs across subgroups. This motivates us to take a bi-level optimisation perspective on fair learning: Optimising the learning strategy based on validation fairness. Specifically, we consider the highly effective workflow of adapting pre-trained models to downstream medical imaging tasks using parameter-efficient fine-tuning (PEFT) techniques. There is a trade-off between updating more parameters, enabling a better fit to the task of interest vs. fewer parameters, potentially reducing the generalisation gap. To manage this tradeoff, we propose FairTune, a framework to optimise the choice of PEFT parameters with respect to fairness. We demonstrate empirically that FairTune leads to improved fairness on a range of medical imaging datasets. The code is available at https://github.com/Raman1121/FairTune