# ColorSwap: A Color and Word Order Dataset for Multimodal Evaluation

arXiv: https://arxiv.org/abs/2402.04492 · HTML: https://mlsanity.com/paper/2402.04492/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 21 references that are in the map.
- First posted on arXiv 7 Feb 2024.

## Builds on (most cited first, 15 of 15)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [FLAVA: A Foundational Language And Vision Alignment Model](https://mlsanity.com/paper/2112.04482/) (2021)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [Simple Open-Vocabulary Object Detection with Vision Transformers](https://mlsanity.com/paper/2205.06230/) (2022)
- [LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding](https://mlsanity.com/paper/2306.17107/) (2023)
- [Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images](https://mlsanity.com/paper/2303.07274/) (2023)
- [The Role of Chain-of-Thought in Complex Vision-Language Reasoning Task](https://mlsanity.com/paper/2311.09193/) (2023)

## Built on by (most cited first, 11 of 11)

- [ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustness](https://mlsanity.com/paper/2504.10514/) (2025)
- [Causal Graphical Models for Vision-Language Compositional Understanding](https://mlsanity.com/paper/2412.09353/) (2024)
- [Test-Time Matching: Unlocking Compositional Reasoning in Multimodal Models](https://mlsanity.com/paper/2510.07632/) (2025)
- [A Good CREPE needs more than just Sugar: Investigating Biases in Compositional Vision-Language Benchmarks](https://mlsanity.com/paper/2506.08227/) (2025)
- [Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal Perspective](https://mlsanity.com/paper/2410.10291/) (2024)
- [Towards Multimodal Active Learning: Efficient Learning with Limited Paired Data](https://mlsanity.com/paper/2510.03247/) (2025)
- [Exploring the Spectrum of Visio-Linguistic Compositionality and Recognition](https://mlsanity.com/paper/2406.09388/) (2024)
- [Diagnosing Vision Language Models' Perception by Leveraging Human Methods for Color Vision Deficiencies](https://mlsanity.com/paper/2505.17461/) (2025)
- [COCO-Tree: Compositional Hierarchical Concept Trees for Enhanced Reasoning in Vision Language Models](https://mlsanity.com/paper/2510.11012/) (2025)
- [Half-Truths Break Similarity-Based Retrieval](https://mlsanity.com/paper/2602.23906/) (2026)
- [Frozen Scenes, Shifting Winners: Configuration Fragility in Text-to-3D Evaluation](https://mlsanity.com/paper/2610.00447/) (2026)

## Abstract (arXiv)

This paper introduces the ColorSwap dataset, designed to assess and improve the proficiency of multimodal models in matching objects with their colors. The dataset is comprised of 2,000 unique image-caption pairs, grouped into 1,000 examples. Each example includes a caption-image pair, along with a ``color-swapped'' pair. We follow the Winoground schema: the two captions in an example have the same words, but the color words have been rearranged to modify different objects. The dataset was created through a novel blend of automated caption and image generation with humans in the loop. We evaluate image-text matching (ITM) and visual language models (VLMs) and find that even the latest ones are still not robust at this task. GPT-4V and LLaVA score 72% and 42% on our main VLM metric, although they may improve with more advanced prompting techniques. On the main ITM metric, contrastive models such as CLIP and SigLIP perform close to chance (at 12% and 30%, respectively), although the non-contrastive BLIP ITM model is stronger (87%). We also find that finetuning on fewer than 2,000 examples yields significant performance gains on this out-of-distribution word-order understanding task. The dataset is here: https://github.com/Top34051/colorswap and here: https://huggingface.co/datasets/stanfordnlp/colorswap.