# VersusDebias: Universal Zero-Shot Debiasing for Text-to-Image Models via SLM-Based Prompt Engineering and Generative Adversary

arXiv: https://arxiv.org/abs/2407.19524 · HTML: https://mlsanity.com/paper/2407.19524/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 52 references that are in the map.
- First posted on arXiv 28 Jul 2024.

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://mlsanity.com/paper/2310.04378/) (2023)
- [On Distillation of Guided Diffusion Models](https://mlsanity.com/paper/2210.03142/) (2022)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022)
- [Unified Concept Editing in Diffusion Models](https://mlsanity.com/paper/2308.14761/) (2023)
- [CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers](https://mlsanity.com/paper/2204.14217/) (2022)
- [SDXL-Lightning: Progressive Adversarial Diffusion Distillation](https://mlsanity.com/paper/2402.13929/) (2024)
- [DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models](https://mlsanity.com/paper/2202.04053/) (2022)
- [Stable Bias: Analyzing Societal Representations in Diffusion Models](https://mlsanity.com/paper/2303.11408/) (2023)
- [Fair Diffusion: Instructing Text-to-Image Generation Models on Fairness](https://mlsanity.com/paper/2302.10893/) (2023)
- [How well can Text-to-Image Generative Models understand Ethical Natural Language Interventions?](https://mlsanity.com/paper/2210.15230/) (2022)
- [Survey of Bias In Text-to-Image Generation: Definition, Evaluation, and Mitigation](https://mlsanity.com/paper/2404.01030/) (2024)
- [Measuring Social Biases in Grounded Vision and Language Embeddings](https://mlsanity.com/paper/2002.08911/) (2020)
- [TIBET: Identifying and Evaluating Biases in Text-to-Image Generative Models](https://mlsanity.com/paper/2312.01261/) (2023)
- [Analyzing Bias in Diffusion-based Face Generation Models](https://mlsanity.com/paper/2305.06402/) (2023)
- [FairRAG: Fair Human Generation via Fair Retrieval Augmentation](https://mlsanity.com/paper/2403.19964/) (2024)
- [SDXS: Real-Time One-Step Latent Diffusion Models with Image Conditions](https://mlsanity.com/paper/2403.16627/) (2024)

## Built on by (most cited first, 6 of 6)

- [AtelierEval: Agentic Evaluation of Humans & LLMs as Text-to-Image Prompters](https://mlsanity.com/paper/2605.22645/) (2026)
- [SynthPert: Enhancing LLM Biological Reasoning via Synthetic Reasoning Traces for Cellular Perturbation Prediction](https://mlsanity.com/paper/2509.25346/) (2025)
- [AutoDebias: Automated Framework for Debiasing Text-to-Image Models](https://mlsanity.com/paper/2508.00445/) (2025)
- [Operationalizing Fairness in Text-to-Image Models: A Survey of Bias, Fairness Audits and Mitigation Strategies](https://mlsanity.com/paper/2604.16516/) (2026)
- [Embedding Arithmetic: A Lightweight, Tuning-Free Framework for Post-hoc Bias Mitigation in Text-to-Image Models](https://mlsanity.com/paper/2604.18167/) (2026)
- [Distributionally Robust Set Representation Learning Under Inference-Time Element Corruption](https://mlsanity.com/paper/2605.30089/) (2026)

## Abstract (arXiv)

With the rapid development of Text-to-Image (T2I) models, biases in human image generation against demographic social groups become a significant concern, impacting fairness and ethical standards in AI. Some researchers propose their methods to tackle with the issue. However, existing methods are designed for specific models with fixed prompts, limiting their adaptability to the fast-evolving models and diverse practical scenarios. Moreover, they neglect the impact of hallucinations, leading to discrepancies between expected and actual results. To address these issues, we introduce VersusDebias, a novel and universal debiasing framework for biases in arbitrary T2I models, consisting of an array generation (AG) module and an image generation (IG) module. The self-adaptive AG module generates specialized attribute arrays to post-process hallucinations and debias multiple attributes simultaneously. The IG module employs a small language model to modify prompts according to the arrays and drives the T2I model to generate debiased images, enabling zero-shot debiasing. Extensive experiments demonstrate VersusDebias's capability to debias any models across gender, race, and age simultaneously. In both zero-shot and few-shot scenarios, VersusDebias outperforms existing methods, showcasing its exceptional utility. Our work is accessible at https://github.com/VersusDebias/VersusDebias to ensure reproducibility and facilitate further research.