DuetMoE: Coupling Inter- and Intra-Subgroup Robustness for Fair Medical Image Analysis

As medical AI expands across diverse healthcare settings worldwide, equitable performance across patient populations is becoming essential to trustworthy clinical use. Fairness in medical image analysis is often evaluated through average performance across predefined subgroups, yet similar subgroup averages can conceal substantial variation among individual patients. Therefore, a reliable medical AI requires addressing two complementary objectives: inter-subgroup fairness, which reduces performance disparities across groups, and intra-subgroup robustness, which protects poorly served patients within each group. To jointly address these objectives, we propose DuetMoE, a subgroup-aware mixture-of-experts framework that couples group-level adaptation with patient-specific clinical guidance, enabling more reliable medical image analysis for individual patients. For settings without linked clinical records, we further introduce DuetMoE+, which retains subgroup-routed experts and incorporates a KL-constrained distributionally robust objective to address intra-subgroup robustness. We evaluate our methods on PI-CAI, radiotherapy, and Harvard-FairSeg. Across four evaluation settings, our methods lead in overall or equity-scaled performance, reduce the inter-subgroup mean Dice gap by up to 30.7%, and raise intra-subgroup 25th-percentile Dice by up to 11.5 points over the strongest reported baselines.