# How Bias Binds: Measuring Hidden Associations for Bias Control in Text-to-Image Compositions

arXiv: https://arxiv.org/abs/2511.07091 · HTML: https://mlsanity.com/paper/2511.07091/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 10 Nov 2025.

## Builds on (most cited first, 26 of 26)

- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [Evaluating Text-to-Visual Generation with Image-to-Text Generation](https://mlsanity.com/paper/2404.01291/) (2024)
- [Unified Concept Editing in Diffusion Models](https://mlsanity.com/paper/2308.14761/) (2023)
- [Stable Bias: Analyzing Societal Representations in Diffusion Models](https://mlsanity.com/paper/2303.11408/) (2023)
- [Debiasing Vision-Language Models via Biased Prompts](https://mlsanity.com/paper/2302.00070/) (2023)
- [How well can Text-to-Image Generative Models understand Ethical Natural Language Interventions?](https://mlsanity.com/paper/2210.15230/) (2022)
- [Finetuning Text-to-Image Diffusion Models for Fairness](https://mlsanity.com/paper/2311.07604/) (2023)
- [No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance](https://mlsanity.com/paper/2404.04125/) (2024)
- [Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting](https://mlsanity.com/paper/1906.09531/) (2019)
- [TIBET: Identifying and Evaluating Biases in Text-to-Image Generative Models](https://mlsanity.com/paper/2312.01261/) (2023)
- [Balancing Act: Distribution-Guided Debiasing in Diffusion Models](https://mlsanity.com/paper/2402.18206/) (2024)
- [FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Composition](https://mlsanity.com/paper/2405.13870/) (2024)
- [FairRAG: Fair Human Generation via Fair Retrieval Augmentation](https://mlsanity.com/paper/2403.19964/) (2024)
- [Balancing the Picture: Debiasing Vision-Language Datasets with Synthetic Contrast Sets](https://mlsanity.com/paper/2305.15407/) (2023)
- [Dissecting and Mitigating Diffusion Bias via Mechanistic Interpretability](https://mlsanity.com/paper/2503.20483/) (2025)
- [Stable Diffusion Exposed: Gender Bias from Prompt to Image](https://mlsanity.com/paper/2312.03027/) (2023)
- [Training Unbiased Diffusion Models From Biased Dataset](https://mlsanity.com/paper/2403.01189/) (2024)
- [Rethinking Training for De-biasing Text-to-Image Generation: Unlocking the Potential of Stable Diffusion](https://mlsanity.com/paper/2408.12692/) (2024)
- [Constrained Diffusion Models via Dual Training](https://mlsanity.com/paper/2408.15094/) (2024)
- [A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks](https://mlsanity.com/paper/2410.07593/) (2024)
- [Towards Transformer-Based Aligned Generation with Self-Coherence Guidance](https://mlsanity.com/paper/2503.17675/) (2025)
- [AIM-Fair: Advancing Algorithmic Fairness via Selectively Fine-Tuning Biased Models with Contextual Synthetic Data](https://mlsanity.com/paper/2503.05665/) (2025)
- [Who Brings the Frisbee: Probing Hidden Hallucination Factors in Large Vision-Language Model via Causality Analysis](https://mlsanity.com/paper/2412.02946/) (2024)
- [Women Wearing Lipstick: Measuring the Bias Between an Object and Its Related Gender](https://mlsanity.com/paper/2310.19130/) (2023)

## Built on by (most cited first, 1 of 1)

- [ContextBias: Controlled Evaluation of Bias Persistence Under Context Shift in Text-to-Image Models](https://mlsanity.com/paper/2608.29847/) (2026)

## Abstract (arXiv)

Text-to-image generative models often exhibit bias related to sensitive attributes. However, current research tends to focus narrowly on single-object prompts with limited contextual diversity. In reality, each object or attribute within a prompt can contribute to bias. For example, the prompt "an assistant wearing a pink hat" may reflect female-inclined biases associated with a pink hat. The neglected joint effects of the semantic binding in the prompts cause significant failures in current debiasing approaches. This work initiates a preliminary investigation on how bias manifests under semantic binding, where contextual associations between objects and attributes influence generative outcomes. We demonstrate that the underlying bias distribution can be amplified based on these associations. Therefore, we introduce a bias adherence score that quantifies how specific object-attribute bindings activate bias. To delve deeper, we develop a training-free context-bias control framework to explore how token decoupling can facilitate the debiasing of semantic bindings. This framework achieves over 10% debiasing improvement in compositional generation tasks. Our analysis of bias scores across various attribute-object bindings and token decorrelation highlights a fundamental challenge: reducing bias without disrupting essential semantic relationships. These findings expose critical limitations in current debiasing approaches when applied to semantically bound contexts, underscoring the need to reassess prevailing bias mitigation strategies.