More is Less? A Simulation-Based Approach to Dynamic Interactions between Biases in Multimodal Models
Not on the map. Why?
Abstract (arXiv)
Multimodal machine learning models, such as those that combine text and image modalities, are increasingly used in critical domains including public safety, security, and healthcare. However, these systems inherit biases from their single modalities. This study proposes a systemic framework for analyzing dynamic multimodal bias interactions. Using the MMBias dataset, which encompasses categories prone to bias such as religion, nationality, and sexual orientation, this study adopts a simulation-based heuristic approach to compute bias scores for text-only, image-only, and multimodal embeddings. A framework is developed to classify bias interactions as amplification (multimodal bias exceeds both unimodal biases), mitigation (multimodal bias is lower than both), and neutrality (multimodal bias lies between unimodal biases), with proportional analyzes conducted to identify the dominant mode and dynamics in these interactions. The findings highlight that amplification (22%) occurs when text and image biases are comparable, while mitigation (11%) arises under the dominance of text bias, highlighting the stabilizing role of image bias. Neutral interactions (67%) are related to a higher text bias without divergence. Conditional probabilities highlight the text's dominance in mitigation and mixed contributions in neutral and amplification cases, underscoring complex modality interplay. In doing so, the study encourages the use of this heuristic, systemic, and interpretable framework to analyze multimodal bias interactions, providing insight into how intermodal biases dynamically interact, with practical applications for multimodal modeling and transferability to context-based datasets, all essential for developing fair and equitable AI models.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 26 of its 51 references that are in the map. First posted on arXiv 23 Dec 2024.
Builds on
26 of 51 references.
References · 26
- 2019A Survey on Bias and Fairness in Machine Learningcited by 1,917
- 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
- 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 296
- 2023Gender bias and stereotypes in Large Language Modelscited by 187
- 2019Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddingscited by 147
- 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 125
- 2022A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learningcited by 104
- 2018Bias and Generalization in Deep Generative Models: An Empirical Studycited by 83
- 2021Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Modelscited by 52
- 2023CLIPSyntel: CLIP and LLM Synergy for Multimodal Question Summarization in Healthcarecited by 43
- 2024Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspectivecited by 39
- 2019Exposing and Correcting the Gender Bias in Image Captioning Datasets and Modelscited by 36
14 more
- 2024CLIP the Bias: How Useful is Balancing Data in Multimodal Learning?cited by 35
- 2020Multimodal Categorization of Crisis Events in Social Mediacited by 34
- 2023Human-Centric Multimodal Machine Learning: Recent Advances and Testbed on AI-based Recruitmentcited by 33
- 2024Review of multimodal machine learning approaches in healthcarecited by 31
- 2024Towards Multimodal Sentiment Analysis Debiasing via Bias Purificationcited by 27
- 2020Analysis of Social Media Data using Multimodal Deep Learning for Disaster Responsecited by 23
- 2020Bias in Multimodal AI: Testbed for Fair Automatic Recruitmentcited by 21
- 2023Multimodal Composite Association Score: Measuring Gender Bias in Generative Multimodal Modelscited by 21
- 2023Safety and Fairness for Content Moderation in Generative Modelscited by 16
- 2022Bias in Machine Learning Models Can Be Significantly Mitigated by Careful Training: Evidence from Neuroimaging Studiescited by 14
- 2024Discover and Mitigate Multiple Biased Subgroups in Image Classifierscited by 11
- 2022Bias and Fairness on Multimodal Emotion Detection Algorithmscited by 10
- 2023Rethinking Multimodal Content Moderation from an Asymmetric Angle with Mixed-modalitycited by 9
- 2023Formalizing Multimedia Recommendation through Multimodal Deep Learningcited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.