# Fair-MoE: Fairness-Oriented Mixture of Experts in Vision-Language Models

arXiv: https://arxiv.org/abs/2502.06094 · HTML: https://mlsanity.com/paper/2502.06094/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 23 of its 45 references that are in the map.
- First posted on arXiv 10 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)
- Mixture-of-experts / [Medical and biomedical MoE](https://mlsanity.com/moe/medical-and-biomedical-moe/)

## Builds on (most cited first, 23 of 23)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [ST-MoE: Designing Stable and Transferable Sparse Expert Models](https://mlsanity.com/paper/2202.08906/) (2022)
- [FairCLIP: Harnessing Fairness in Vision-Language Learning](https://mlsanity.com/paper/2403.19949/) (2024)
- [Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Study](https://mlsanity.com/paper/2209.15517/) (2022)
- [ELIXR: Towards a general purpose X-ray artificial intelligence system through alignment of large language models and radiology vision encoders](https://mlsanity.com/paper/2308.01317/) (2023)
- [MedCoT: Medical Chain of Thought via Hierarchical Expert](https://mlsanity.com/paper/2412.13736/) (2024)
- [A ChatGPT Aided Explainable Framework for Zero-Shot Medical Image Diagnosis](https://mlsanity.com/paper/2307.01981/) (2023)
- [Addressing Fairness Issues in Deep Learning-Based Medical Image Analysis: A Systematic Review](https://mlsanity.com/paper/2209.13177/) (2022)
- [Debiasing Methods for Fairer Neural Models in Vision and Language Research: A Survey](https://mlsanity.com/paper/2211.05617/) (2022)
- [VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Framework](https://mlsanity.com/paper/2403.09027/) (2024)
- [Cross-token Modeling with Conditional Computation](https://mlsanity.com/paper/2109.02008/) (2021)
- [WorldGPT: A Sora-Inspired Video AI Agent as Rich World Models from Text and Image Inputs](https://mlsanity.com/paper/2403.07944/) (2024)
- [MedThink: Explaining Medical Visual Question Answering via Multimodal Decision-Making Rationale](https://mlsanity.com/paper/2404.12372/) (2024)
- [KPL: Training-Free Medical Knowledge Mining of Vision-Language Models](https://mlsanity.com/paper/2501.11231/) (2025)
- [Fairness-aware Vision Transformer via Debiased Self-Attention](https://mlsanity.com/paper/2301.13803/) (2023)
- [FEAMOE: Fair, Explainable and Adaptive Mixture of Experts](https://mlsanity.com/paper/2210.04995/) (2022)
- [FairViT: Fair Vision Transformer via Adaptive Masking](https://mlsanity.com/paper/2407.14799/) (2024)
- [VisionGPT-3D: A Generalized Multimodal Agent for Enhanced 3D Vision Understanding](https://mlsanity.com/paper/2403.09530/) (2024)

## Built on by (most cited first, 6 of 6)

- [3D-RAD: A Comprehensive 3D Radiology Med-VQA Dataset with Multi-Temporal Analysis and Diverse Diagnostic Tasks](https://mlsanity.com/paper/2506.11147/) (2025)
- [Med-GLIP: Advancing Medical Language-Image Pre-training with Large-scale Grounded Dataset](https://mlsanity.com/paper/2508.10528/) (2025)
- [Self-Calibrated Consistency can Fight Back for Adversarial Robustness in Vision-Language Models](https://mlsanity.com/paper/2510.22785/) (2025)
- [Equinox: Holistic Fair Scheduling in Serving Large Language Models](https://mlsanity.com/paper/2508.16646/) (2025)
- [Developing Fairness-Aware Task Decomposition to Improve Equity in Post-Spinal Fusion Complication Prediction](https://mlsanity.com/paper/2512.00598/) (2025)
- [FairLLaVA: Fairness-Aware Parameter-Efficient Fine-Tuning for Large Vision-Language Assistants](https://mlsanity.com/paper/2603.26008/) (2026)

## Abstract (arXiv)

Fairness is a fundamental principle in medical ethics. Vision Language Models (VLMs) have shown significant potential in the medical field due to their ability to leverage both visual and linguistic contexts, reducing the need for large datasets and enabling the performance of complex tasks. However, the exploration of fairness within VLM applications remains limited. Applying VLMs without a comprehensive analysis of fairness could lead to concerns about equal treatment opportunities and diminish public trust in medical deep learning models. To build trust in medical VLMs, we propose Fair-MoE, a model specifically designed to ensure both fairness and effectiveness. Fair-MoE comprises two key components: textitthe Fairness-Oriented Mixture of Experts (FO-MoE) and textitthe Fairness-Oriented Loss (FOL). FO-MoE is designed to leverage the expertise of various specialists to filter out biased patch embeddings and use an ensemble approach to extract more equitable information relevant to specific tasks. FOL is a novel fairness-oriented loss function that not only minimizes the distances between different attributes but also optimizes the differences in the dispersion of various attributes' distributions. Extended experiments demonstrate the effectiveness and fairness of Fair-MoE. Tested on the Harvard-FairVLMed dataset, Fair-MoE showed improvements in both fairness and accuracy across all four attributes. Code will be publicly available.