# VLSlice: Interactive Vision-and-Language Slice Discovery

arXiv: https://arxiv.org/abs/2309.06703 · HTML: https://mlsanity.com/paper/2309.06703/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 42 references that are in the map.
- First posted on arXiv 13 Sep 2023.

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [Multimodal datasets: misogyny, pornography, and malignant stereotypes](https://mlsanity.com/paper/2110.01963/) (2021)
- [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://mlsanity.com/paper/1906.08976/) (2019)
- [12-in-1: Multi-Task Vision and Language Representation Learning](https://mlsanity.com/paper/1912.02315/) (2019)
- [Improving fairness in machine learning systems: What do industry practitioners need?](https://mlsanity.com/paper/1812.05239/) (2018)
- [Women also Snowboard: Overcoming Bias in Captioning Models](https://mlsanity.com/paper/1803.09797/) (2018)
- [Multiaccuracy: Black-Box Post-Processing for Fairness in Classification](https://mlsanity.com/paper/1805.12317/) (2018)
- [No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems](https://mlsanity.com/paper/2011.12945/) (2020)
- [Domino: Discovering Systematic Errors with Cross-Modal Embeddings](https://mlsanity.com/paper/2203.14960/) (2022)
- [FairVis: Visual Analytics for Discovering Intersectional Bias in Machine Learning](https://mlsanity.com/paper/1904.05419/) (2019)
- [The Spotlight: A General Method for Discovering Systematic Errors in Deep Learning Models](https://mlsanity.com/paper/2107.00758/) (2021)
- [Measuring Social Biases in Grounded Vision and Language Embeddings](https://mlsanity.com/paper/2002.08911/) (2020)
- [Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Models](https://mlsanity.com/paper/2104.08666/) (2021)
- [Quantifying Societal Bias Amplification in Image Captioning](https://mlsanity.com/paper/2203.15395/) (2022)
- [It is Okay to Not Be Okay: Overcoming Emotional Bias in Affective Image Captioning by Contrastive Data Collection](https://mlsanity.com/paper/2204.07660/) (2022)
- [Measuring Representational Harms in Image Captioning](https://mlsanity.com/paper/2206.07173/) (2022)
- [Slice-based Learning: A Programming Model for Residual Learning in Critical Data Slices](https://mlsanity.com/paper/1909.06349/) (2019)
- [Human-in-the-loop Extraction of Interpretable Concepts in Deep Learning Models](https://mlsanity.com/paper/2108.03738/) (2021)
- [Slice Tuner: A Selective Data Acquisition Framework for Accurate and Fair Machine Learning Models](https://mlsanity.com/paper/2003.04549/) (2020)

## Built on by (most cited first, 7 of 7)

- [Adversarial Nibbler: An Open Red-Teaming Method for Identifying Diverse Harms in Text-to-Image Generation](https://mlsanity.com/paper/2403.12075/) (2024)
- [HiBug2: Efficient and Interpretable Error Slice Discovery for Comprehensive Model Debugging](https://mlsanity.com/paper/2501.16751/) (2025)
- [Unearthing Skill-Level Insights for Understanding Trade-Offs of Foundation Models](https://mlsanity.com/paper/2410.13826/) (2024)
- [SPIN: Hierarchical Segmentation with Subpart Granularity in Natural Images](https://mlsanity.com/paper/2407.09686/) (2024)
- [Debiasing Vison-Language Models with Text-Only Training](https://mlsanity.com/paper/2410.09365/) (2024)
- [Detecting Systematic Weaknesses in Vision Models along Predefined Human-Understandable Dimensions](https://mlsanity.com/paper/2502.12360/) (2025)
- [LatentDiff: Scaling Semantic Dataset Comparison to Millions of Images](https://mlsanity.com/paper/2605.00899/) (2026)

## Abstract (arXiv)

Recent work in vision-and-language demonstrates that large-scale pretraining can learn generalizable models that are efficiently transferable to downstream tasks. While this may improve dataset-scale aggregate metrics, analyzing performance around hand-crafted subgroups targeting specific bias dimensions reveals systemic undesirable behaviors. However, this subgroup analysis is frequently stalled by annotation efforts, which require extensive time and resources to collect the necessary data. Prior art attempts to automatically discover subgroups to circumvent these constraints but typically leverages model behavior on existing task-specific annotations and rapidly degrades on more complex inputs beyond "tabular" data, none of which study vision-and-language models. This paper presents VLSlice, an interactive system enabling user-guided discovery of coherent representation-level subgroups with consistent visiolinguistic behavior, denoted as vision-and-language slices, from unlabeled image sets. We show that VLSlice enables users to quickly generate diverse high-coherency slices in a user study (n=22) and release the tool publicly.