# Evaluating Text-to-Image Matching using Binary Image Selection (BISON)

arXiv: https://arxiv.org/abs/1901.06595 · HTML: https://mlsanity.com/paper/1901.06595/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jan 2019.

## Built on by (most cited first, 12 of 12)

- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [CREPE: Can Vision-Language Foundation Models Reason Compositionally?](https://mlsanity.com/paper/2212.07796/) (2022)
- [The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editing](https://mlsanity.com/paper/2311.01410/) (2023)
- [Equivariant Similarity for Vision-Language Foundation Models](https://mlsanity.com/paper/2303.14465/) (2023)
- [Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Models](https://mlsanity.com/paper/2308.16463/) (2023)
- [Progressive Compositionality in Text-to-Image Generative Models](https://mlsanity.com/paper/2410.16719/) (2024)
- [Do Vision-and-Language Transformers Learn Grounded Predicate-Noun Dependencies?](https://mlsanity.com/paper/2210.12079/) (2022)
- [Comparison Visual Instruction Tuning](https://mlsanity.com/paper/2406.09240/) (2024)
- [Uncertainty-based Cross-Modal Retrieval with Probabilistic Representations](https://mlsanity.com/paper/2204.09268/) (2022)
- [A Universal Model for Cross Modality Mapping by Relational Reasoning](https://mlsanity.com/paper/2102.13360/) (2021)
- [DM-Align: Leveraging the Power of Natural Language Instructions to Make Changes to Images](https://mlsanity.com/paper/2404.18020/) (2024)

## Abstract (arXiv)

Providing systems the ability to relate linguistic and visual content is one of the hallmarks of computer vision. Tasks such as text-based image retrieval and image captioning were designed to test this ability but come with evaluation measures that have a high variance or are difficult to interpret. We study an alternative task for systems that match text and images: given a text query, the system is asked to select the image that best matches the query from a pair of semantically similar images. The system's accuracy on this Binary Image SelectiON (BISON) task is interpretable, eliminates the reliability problems of retrieval evaluations, and focuses on the system's ability to understand fine-grained visual structure. We gather a BISON dataset that complements the COCO dataset and use it to evaluate modern text-based image retrieval and image captioning systems. Our results provide novel insights into the performance of these systems. The COCO-BISON dataset and corresponding evaluation code are publicly available from urlhttp://hexianghu.com/bison/.