mlsanityData to 7 Oct 2026

Improving Fairness using Vision-Language Driven Image Augmentation

arXiv 2311.01573 ↗First posted 2 Nov 2023cs.CV cs.AI cs.CYcited by 13

Not on the map. Why?

Abstract (arXiv)

Fairness is crucial when training a deep-learning discriminative model, especially in the facial domain. Models tend to correlate specific characteristics (such as age and skin color) with unrelated attributes (downstream tasks), resulting in biases which do not correspond to reality. It is common knowledge that these correlations are present in the data and are then transferred to the models during training. This paper proposes a method to mitigate these correlations to improve fairness. To do so, we learn interpretable and meaningful paths lying in the semantic space of a pre-trained diffusion model (DiffAE) -- such paths being supervised by contrastive text dipoles. That is, we learn to edit protected characteristics (age and skin color). These paths are then applied to augment images to improve the fairness of a given dataset. We test the proposed method on CelebA-HQ and UTKFace on several downstream tasks with age and skin color as protected characteristics. As a proxy for fairness, we compute the difference in accuracy with respect to the protected characteristics. Quantitative results show how the augmented images help the model improve the overall accuracy, the aforementioned metric, and the disparity of equal opportunity. Code is available at: https://github.com/Moreno98/Vision-Language-Bias-Control.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 29 of its 45 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

29 of 45 references.

References · 29

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2020Denoising Diffusion Implicit Modelscited by 9,655
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  5. 2022Generative Adversarial Networkscited by 5,603
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  7. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  8. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  9. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  10. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 447
  11. 2017Age Progression/Regression by Conditional Adversarial Autoencodercited by 426
  12. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
17 more
  1. 2019Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigationcited by 252
  2. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  3. 2018FairGAN: Fairness-aware Generative Adversarial Networkscited by 164
  4. 2020Challenges in Deploying Machine Learning: a Survey of Case Studiescited by 151
  5. 2020Learning from Failure: Training Debiased Classifier from Biased Classifiercited by 133
  6. 2020Evaluating and Mitigating Bias in Image Classifiers: A Causal Perspective Using Counterfactualscited by 56
  7. 2021WarpedGANSpace: Finding non-linear RBF paths in GAN latent spacecited by 48
  8. 2023HyperReenact: One-Shot Reenactment via Jointly Learning to Refine and Retarget Facescited by 48
  9. 2023Attribute-preserving Face Dataset Anonymization via Latent Code Optimizationcited by 43
  10. 2021Learning Fair Classifiers with Partially Annotated Group Labelscited by 40
  11. 2020Intra-Processing Methods for Debiasing Neural Networkscited by 27
  12. 2022PandA: Unsupervised Learning of Parts and Appearances in the Feature Maps of GANscited by 20
  13. 2022StyleMask: Disentangling the Style Space of StyleGAN2 for Neural Face Reenactmentcited by 17
  14. 2021Tensor Component Analysis for Interpreting the Latent Space of GANscited by 13
  15. 2022ContraCLIP: Interpretable GAN generation driven by pairs of contrasting sentencescited by 12
  16. 2021Does Data Repair Lead to Fair Models? Curating Contextually Fair Data To Reduce Model Biascited by 11
  17. 2023Parts of Speech-Grounded Subspaces in Vision-Language Modelscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 2; 2026: 2.

13 papers.

Sort:

By area:

  1. 2024OpenBias: Open-set Bias Detection in Text-to-Image Generative Modelscited by 61
  2. 2024Are CLIP features all you need for Universal Synthetic Image Origin Attribution?cited by 14
  3. 2024MIST: Mitigating Intersectional Bias with Disentangled Cross-Attention Editing in Text-to-Image Diffusion Modelscited by 10
  4. 2024MM2Latent: Text-to-facial image generation and editing in GANs with multimodal assistancecited by 7
  5. 2025AIM-Fair: Advancing Algorithmic Fairness via Selectively Fine-Tuning Biased Models with Contextual Synthetic Datacited by 5
  6. 2024Trustworthy AI-Generative Content for Intelligent Network Service: Robustness, Security, and Fairnesscited by 3
  7. 2024GradBias: Unveiling Word Influence on Bias in Text-to-Image Generative Modelscited by 2
  8. 2024On the Fairness, Diversity and Reliability of Text-to-Image Generative Modelscited by 2
  9. 2024Fairer Analysis and Demographically Balanced Face Generation for Fairer Face Verificationcited by 2
  10. 2025VisMoDAl: Visual Analytics for Evaluating and Improving Corruption Robustness of Vision-Language Modelscited by 1
  11. 2024Toward Fairer Face Recognition Datasetscited by 0
  12. 2026ProtoFair: Fair Self-Supervised Contrastive Learning via Pseudo-Counterfactual Pairscited by 0
1 more
  1. 2026Do Counterfactually Fair Image Classifiers Satisfy Group Fairness? -- A Theoretical and Empirical Studycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Debias Anything: Fairness with Diversity without Supervision in Diffusion Models
  2. 2023Leveraging Diffusion Perturbations for Measuring Fairness in Computer Vision
  3. 2024Evaluating Fairness in Large Vision-Language Models Across Diverse Demographic Attributes and Prompts
  4. 2024Configurable Fairness: Direct Optimization of Parity Metrics via Vision-Language Models
  5. 2025BioPro: Towards Difference-Aware Gender Fairness for Vision-Language Models
  6. 2026Benchmarking Bias Mitigation Toward Fairness Without Harm from Vision to LVLMs
  7. 2023Fair Text-to-Image Diffusion via Fair Mapping
  8. 2026A Closed-Form Solution for Debiasing Vision-Language Models with Utility Guarantees Across Modalities and Tasks