# Mass-Producing Failures of Multimodal Systems with Language Models

arXiv: https://arxiv.org/abs/2306.12105 · HTML: https://mlsanity.com/paper/2306.12105/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 124 references that are in the map.
- First posted on arXiv 21 Jun 2023.

## Builds on (most cited first, 30 of 55)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [Release Strategies and the Social Impacts of Language Models](https://mlsanity.com/paper/1908.09203/) (2019)
- [On Measuring Social Biases in Sentence Encoders](https://mlsanity.com/paper/1903.10561/) (2019)
- [Shap-E: Generating Conditional 3D Implicit Functions](https://mlsanity.com/paper/2305.02463/) (2023)
- [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021)
- [Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery](https://mlsanity.com/paper/2302.03668/) (2023)
- [VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2303.08320/) (2023)

## Built on by (most cited first, 30 of 36)

- [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://mlsanity.com/paper/2406.16860/) (2024)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training](https://mlsanity.com/paper/2501.17161/) (2025)
- [MetaMorph: Multimodal Understanding and Generation via Instruction Tuning](https://mlsanity.com/paper/2412.14164/) (2024)
- [Foundational Models Defining a New Era in Vision: A Survey and Outlook](https://mlsanity.com/paper/2307.13721/) (2023)
- [How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs](https://mlsanity.com/paper/2311.16101/) (2023)
- [Red-Teaming for Generative AI: Silver Bullet or Security Theater?](https://mlsanity.com/paper/2401.15897/) (2024)
- [Visual Hallucinations of Multi-modal Large Language Models](https://mlsanity.com/paper/2402.14683/) (2024)
- [TokenCompose: Text-to-Image Diffusion with Token-level Supervision](https://mlsanity.com/paper/2312.03626/) (2023)
- [Understanding and Improving Training-free Loss-based Diffusion Guidance](https://mlsanity.com/paper/2403.12404/) (2024)
- [Adversaries Can Misuse Combinations of Safe Models](https://mlsanity.com/paper/2406.14595/) (2024)
- [What Factors Affect Multi-Modal In-Context Learning? An In-Depth Exploration](https://mlsanity.com/paper/2410.20482/) (2024)
- [CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modally](https://mlsanity.com/paper/2502.03566/) (2025)
- [Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filters](https://mlsanity.com/paper/2403.02677/) (2024)
- [MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models](https://mlsanity.com/paper/2409.15477/) (2024)
- [Effective Human-AI Teams via Learned Natural Language Rules and Onboarding](https://mlsanity.com/paper/2311.01007/) (2023)
- [A Philosophical Introduction to Language Models - Part II: The Way Forward](https://mlsanity.com/paper/2405.03207/) (2024)
- [Anytime Continual Learning for Open Vocabulary Classification](https://mlsanity.com/paper/2409.08518/) (2024)
- [Evaluating Vision-Language Models as Evaluators in Path Planning](https://mlsanity.com/paper/2411.18711/) (2024)
- [What's Producible May Not Be Reachable: Measuring the Steerability of Generative Models](https://mlsanity.com/paper/2503.17482/) (2025)
- [ASPIRE: Language-Guided Data Augmentation for Improving Robustness Against Spurious Correlations](https://mlsanity.com/paper/2308.10103/) (2023)
- [Identifying Systematic Errors in Object Detectors with the SCROD Pipeline](https://mlsanity.com/paper/2309.13489/) (2023)
- [Uncovering Gaps in How Humans and LLMs Interpret Subjective Language](https://mlsanity.com/paper/2503.04113/) (2025)
- [GHOST: Hallucination-Inducing Image Generation for Multimodal LLMs](https://mlsanity.com/paper/2509.25178/) (2025)
- [Sound Check: Auditing Audio Datasets](https://mlsanity.com/paper/2410.13114/) (2024)
- [Discovering Divergent Representations between Text-to-Image Models](https://mlsanity.com/paper/2509.08940/) (2025)
- [Unveiling Glitches: A Deep Dive into Image Encoding Bugs within CLIP](https://mlsanity.com/paper/2407.00592/) (2024)
- [Uncovering the Computational Ingredients of Human-Like Representations in LLMs](https://mlsanity.com/paper/2510.01030/) (2025)
- [All-in-One Conditioning for Text-to-Image Synthesis](https://mlsanity.com/paper/2602.09165/) (2026)
- [Multi-Modal Learning meets Genetic Programming: Analyzing Alignment in Latent Space Optimization](https://mlsanity.com/paper/2604.08324/) (2026)

## Abstract (arXiv)

Deployed multimodal systems can fail in ways that evaluators did not anticipate. In order to find these failures before deployment, we introduce MultiMon, a system that automatically identifies systematic failures -- generalizable, natural-language descriptions of patterns of model failures. To uncover systematic failures, MultiMon scrapes a corpus for examples of erroneous agreement: inputs that produce the same output, but should not. It then prompts a language model (e.g., GPT-4) to find systematic patterns of failure and describe them in natural language. We use MultiMon to find 14 systematic failures (e.g., "ignores quantifiers") of the CLIP text-encoder, each comprising hundreds of distinct inputs (e.g., "a shelf with a few/many books"). Because CLIP is the backbone for most state-of-the-art multimodal systems, these inputs produce failures in Midjourney 5.1, DALL-E, VideoFusion, and others. MultiMon can also steer towards failures relevant to specific use cases, such as self-driving cars. We see MultiMon as a step towards evaluation that autonomously explores the long tail of potential system failures. Code for MULTIMON is available at https://github.com/tsb0601/MultiMon.