# LightFair: Towards an Efficient Alternative for Fair T2I Diffusion via Debiasing Pre-trained Text Encoders

arXiv: https://arxiv.org/abs/2509.23639 · HTML: https://mlsanity.com/paper/2509.23639/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- First posted on arXiv 28 Sep 2025.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Built on by (most cited first, 16 of 16)

- [RAPO++: Cross-Stage Prompt Optimization for Text-to-Video Generation via Data Alignment and Test-Time Scaling](https://mlsanity.com/paper/2510.20206/) (2025)
- [Bias Is a Subspace, Not a Coordinate: A Geometric Rethinking of Post-hoc Debiasing in Vision-Language Models](https://mlsanity.com/paper/2511.18123/) (2025)
- [Making Training-Free Diffusion Segmentors Scale with the Generative Power](https://mlsanity.com/paper/2603.06178/) (2026)
- [Breaking Dual Bottlenecks: Evolving Unified Multimodal Models into Self-Adaptive Interleaved Visual Reasoners](https://mlsanity.com/paper/2605.14709/) (2026)
- [Quantifying the Potential to Escape Filter Bubbles: A Behavior-Aware Measure via Contrastive Simulation](https://mlsanity.com/paper/2512.03067/) (2025)
- [Guiding Diffusion-based Reconstruction with Contrastive Signals for Balanced Visual Representation](https://mlsanity.com/paper/2603.04803/) (2026)
- [Stay Fair! Ensuring Group Fairness in Diffusion Models Across Guidance Scales](https://mlsanity.com/paper/2605.28036/) (2026)
- [Position: Universal Aesthetic Alignment Narrows Artistic Expression](https://mlsanity.com/paper/2512.11883/) (2025)
- [Mind the Way You Select Negative Texts: Pursuing the Distance Consistency in OOD Detection with VLMs](https://mlsanity.com/paper/2603.02618/) (2026)
- [BlackMirror: Black-Box Backdoor Detection for Text-to-Image Models via Instruction-Response Deviation](https://mlsanity.com/paper/2603.05921/) (2026)
- [HoloFair: Unified T2I Fairness Evaluation and Fair-GRPO Debiasing](https://mlsanity.com/paper/2605.24687/) (2026)
- [Understanding-Enhanced Model Collaboration for Long-Tailed Egocentric Mistake Detection](https://mlsanity.com/paper/2606.02120/) (2026)
- [Training-Free Debiasing of Diffusion Models via CLIP-Guided Denoising Optimization](https://mlsanity.com/paper/2607.00817/) (2026)
- [HiGFA: Hierarchical Guidance for Fine-grained Data Augmentation with Diffusion Models](https://mlsanity.com/paper/2511.12547/) (2025)
- [EquiSteer: Cross-Attention Steering Towards a Fairer Text-Guided Image Generation](https://mlsanity.com/paper/2607.01147/) (2026)
- [FairFlow: Demystifying and Mitigating Stereotype Bias in Text-to-Image Diffusion Transformers](https://mlsanity.com/paper/2607.03180/) (2026)

## Abstract (arXiv)

This paper explores a novel lightweight approach LightFair to achieve fair text-to-image diffusion models (T2I DMs) by addressing the adverse effects of the text encoder. Most existing methods either couple different parts of the diffusion model for full-parameter training or rely on auxiliary networks for correction. They incur heavy training or sampling burden and unsatisfactory performance. Since T2I DMs consist of multiple components, with the text encoder being the most fine-tunable and front-end module, this paper focuses on mitigating bias by fine-tuning text embeddings. To validate feasibility, we observe that the text encoder's neutral embedding output shows substantial skewness across image embeddings of various attributes in the CLIP space. More importantly, the noise prediction network further amplifies this imbalance. To finetune the text embedding, we propose a collaborative distance-constrained debiasing strategy that balances embedding distances to improve fairness without auxiliary references. However, mitigating bias can compromise the original generation quality. To address this, we introduce a two-stage text-guided sampling strategy to limit when the debiased text encoder intervenes. Extensive experiments demonstrate that LightFair is effective and efficient. Notably, on Stable Diffusion v1.5, our method achieves SOTA debiasing at just 1/4 of the training burden, with virtually no increase in sampling burden. The code is available at https://github.com/boyuh/LightFair.