# Chameleon: Foundation Models for Fairness-aware Multi-modal Data Augmentation to Enhance Coverage of Minorities

arXiv: https://arxiv.org/abs/2402.01071 · HTML: https://mlsanity.com/paper/2402.01071/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 98 references that are in the map.
- First posted on arXiv 2 Feb 2024.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Lumiere: A Space-Time Diffusion Model for Video Generation](https://mlsanity.com/paper/2401.12945/) (2024)
- [Age Progression/Regression by Conditional Adversarial Autoencoder](https://mlsanity.com/paper/1702.08423/) (2017)
- [NIMA: Neural Image Assessment](https://mlsanity.com/paper/1709.05424/) (2017)
- [Real-valued (Medical) Time Series Generation with Recurrent Conditional GANs](https://mlsanity.com/paper/1706.02633/) (2017)
- [Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoning](https://mlsanity.com/paper/2301.13808/) (2023)
- [Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes](https://mlsanity.com/paper/2304.09433/) (2023)
- [Model Patching: Closing the Subgroup Performance Gap with Data Augmentation](https://mlsanity.com/paper/2008.06775/) (2020)
- [How to Prompt LLMs for Text-to-SQL: A Study in Zero-shot, Single-domain, and Cross-domain Settings](https://mlsanity.com/paper/2305.11853/) (2023)
- [ImDiffusion: Imputed Diffusion Models for Multivariate Time Series Anomaly Detection](https://mlsanity.com/paper/2307.00754/) (2023)
- [Data augmentation using synthetic data for time series classification with deep residual networks](https://mlsanity.com/paper/1808.02455/) (2018)
- [Representation Bias in Data: A Survey on Identification and Resolution Techniques](https://mlsanity.com/paper/2203.11852/) (2022)
- [Assessing and Remedying Coverage for a Given Dataset](https://mlsanity.com/paper/1810.06742/) (2018)
- [Relational Data Synthesis using Generative Adversarial Networks: A Design Space Exploration](https://mlsanity.com/paper/2008.12763/) (2020)
- [PrivLava: Synthesizing Relational Data with Foreign Keys under Differential Privacy](https://mlsanity.com/paper/2304.04545/) (2023)
- [Data Cleaning for Accurate, Fair, and Robust Models: A Big Data - AI Integration Approach](https://mlsanity.com/paper/1904.10761/) (2019)
- [Through the Fairness Lens: Experimental Analysis and Evaluation of Entity Matching](https://mlsanity.com/paper/2307.02726/) (2023)
- [Fairness and Bias in Truth Discovery Algorithms: An Experimental Analysis](https://mlsanity.com/paper/2304.12573/) (2023)

## Built on by (most cited first, 5 of 5)

- [MedEqualizer: A Framework Investigating Bias in Synthetic Medical Data and Mitigation via Augmentation](https://mlsanity.com/paper/2511.01054/) (2025)
- [CounterFace: A Synthetic Face Dataset for Fine-Grained Counterfactual Evaluation of Face Recognition Systems](https://mlsanity.com/paper/2407.13922/) (2024)
- [Unbiased Binning for Fairness-aware Attribute Representation](https://mlsanity.com/paper/2509.21785/) (2025)
- [Be Fair! Can Machine Learning Engineering Agents Adhere to Fairness Constraints?](https://mlsanity.com/paper/2606.04971/) (2026)
- [Stellar: Scalable Multimodal Document Retrieval for Natural Language Queries](https://mlsanity.com/paper/2606.19960/) (2026)

## Abstract (arXiv)

The potential harms of the under-representation of minorities in training data, particularly in multi-modal settings, is a well-recognized concern. While there has been extensive effort in detecting such under-representation, resolution has remained a challenge. With recent advancements in generative AI, large language models and foundation models have emerged as versatile tools across various domains. In this paper, we propose Chameleon, a system that efficiently utilizes these tools to augment a data set with a minimal addition of synthetically generated tuples, in order to enhance the coverage of the under-represented groups. Our system follows a rejection sampling approach to ensure the generated tuples have a high quality and follow the underlying distribution. In order to minimize the rejection chance of the generated tuples, we propose multiple strategies for providing a guide for the foundation model. Our experiment results, in addition to confirming the efficiency of our proposed algorithms, illustrate the effectiveness of our approach, as the unfairness of the model in a downstream task significantly dropped after data repair using Chameleon.