# SHaSaM: Submodular Hard Sample Mining for Fair Facial Attribute Recognition

arXiv: https://arxiv.org/abs/2602.05162 · HTML: https://mlsanity.com/paper/2602.05162/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 68 references that are in the map.
- First posted on arXiv 5 Feb 2026.

## Builds on (most cited first, 30 of 55)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Deep Metric Learning via Lifted Structured Feature Embedding](https://mlsanity.com/paper/1511.06452/) (2015)
- [Age Progression/Regression by Conditional Adversarial Autoencoder](https://mlsanity.com/paper/1702.08423/) (2017)
- [Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints](https://mlsanity.com/paper/1707.09457/) (2017)
- [InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs](https://mlsanity.com/paper/2005.09635/) (2020)
- [Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning](https://mlsanity.com/paper/1904.06627/) (2019)
- [Learning Adversarially Fair and Transferable Representations](https://mlsanity.com/paper/1802.06309/) (2018)
- [Learning Not to Learn: Training Deep Neural Networks with Biased Data](https://mlsanity.com/paper/1812.10352/) (2018)
- [Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation](https://mlsanity.com/paper/1911.11834/) (2019)
- [Flexibly Fair Representation Learning by Disentanglement](https://mlsanity.com/paper/1906.02589/) (2019)
- [Fair Attribute Classification through Latent Space De-biasing](https://mlsanity.com/paper/2012.01469/) (2020)
- [Lazier Than Lazy Greedy](https://mlsanity.com/paper/1409.7938/) (2014)
- [Analyzing and Improving Representations with the Soft Nearest Neighbor Loss](https://mlsanity.com/paper/1902.01889/) (2019)
- [Submodular Combinatorial Information Measures with Applications in Machine Learning](https://mlsanity.com/paper/2006.15412/) (2020)
- [Achieving Fairness through Adversarial Learning: an Application to Recidivism Prediction](https://mlsanity.com/paper/1807.00199/) (2018)
- [Fairness by Learning Orthogonal Disentangled Representations](https://mlsanity.com/paper/2003.05707/) (2020)
- [Fair Contrastive Learning for Facial Attribute Classification](https://mlsanity.com/paper/2203.16209/) (2022)
- [Learning From Less Data: A Unified Data Subset Selection and Active Learning Framework for Computer Vision](https://mlsanity.com/paper/1901.01151/) (2019)
- [Fair Feature Distillation for Visual Recognition](https://mlsanity.com/paper/2106.04411/) (2021)

## Built on by (most cited first, 1 of 1)

- [Fairness-Aware Mixture-of-Experts via Subgroup Reweighting and Gate Regularization](https://mlsanity.com/paper/2608.22820/) (2026)

## Abstract (arXiv)

Deep neural networks often inherit social and demographic biases from annotated data during model training, leading to unfair predictions, especially in the presence of sensitive attributes like race, age, gender etc. Existing methods fall prey to the inherent data imbalance between attribute groups and inadvertently emphasize on sensitive attributes, worsening unfairness and performance. To surmount these challenges, we propose SHaSaM (Submodular Hard Sample Mining), a novel combinatorial approach that models fairness-driven representation learning as a submodular hard-sample mining problem. Our two-stage approach comprises of SHaSaM-MINE, which introduces a submodular subset selection strategy to mine hard positives and negatives - effectively mitigating data imbalance, and SHaSaM-LEARN, which introduces a family of combinatorial loss functions based on Submodular Conditional Mutual Information to maximize the decision boundary between target classes while minimizing the influence of sensitive attributes. This unified formulation restricts the model from learning features tied to sensitive attributes, significantly enhancing fairness without sacrificing performance. Experiments on CelebA and UTKFace demonstrate that SHaSaM achieves state-of-the-art results, with up to 2.7 points improvement in model fairness (Equalized Odds) and a 3.5% gain in Accuracy, within fewer epochs as compared to existing methods.