Unveiling Glitches: A Deep Dive into Image Encoding Bugs within CLIP
Not on the map. Why?
Abstract (arXiv)
Understanding the limitations and weaknesses of state-of-the-art models in artificial intelligence is crucial for their improvement and responsible application. In this research, we focus on CLIP, a model renowned for its integration of vision and language processing. Our objective is to uncover recurring problems and blind spots in CLIP's image comprehension. By delving into both the commonalities and disparities between CLIP and human image understanding, we augment our comprehension of these models' capabilities. Through our analysis, we reveal significant discrepancies in CLIP's interpretation of images compared to human perception, shedding light on areas requiring improvement. Our methodologies, the Discrepancy Analysis Framework (DAF) and the Transformative Caption Analysis for CLIP (TCAC), enable a comprehensive evaluation of CLIP's performance. We identify 14 systemic faults, including Action vs. Stillness confusion, Failure to identify the direction of movement or positioning of objects in the image, Hallucination of Water-like Features, Misattribution of Geographic Context, among others. By addressing these limitations, we lay the groundwork for the development of more accurate and nuanced image embedding models, contributing to advancements in artificial intelligence.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 11 of its 12 references that are in the map. First posted on arXiv 30 Jun 2024.
Builds on
11 of 12 references.
References · 11
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
- 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
- 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
- 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
- 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
- 2023A Comprehensive Survey on Vector Database: Storage and Retrieval Technique, Challengecited by 55
- 2022Discovering Bugs in Vision Models using Off-the-shelf Image Generation and Captioningcited by 45
- 2023Mass-Producing Failures of Multimodal Systems with Language Modelscited by 36
Built on by
Citing papers per year: 2025: 1.
1 paper.