One Single Hub Text Breaks CLIP: Identifying Vulnerabilities in Cross-Modal Encoders via Hubness
Not on the map. Why?
Abstract (arXiv)
The hubness problem, in which hub embeddings are close to many unrelated examples, occurs often in high-dimensional embedding spaces and may pose a practical threat for purposes such as information retrieval and automatic evaluation metrics. In particular, since cross-modal similarity between text and images cannot be calculated by direct comparisons, such as string matching, cross-modal encoders that project different modalities into a shared space are helpful for various cross-modal applications, and thus, the existence of hubs may pose practical threats. To reveal the vulnerabilities of cross-modal encoders, we propose a method for identifying the hub embedding and its corresponding hub text. Experiments on image captioning evaluation in MSCOCO and nocaps along with image-to-text retrieval tasks in MSCOCO and Flickr30k showed that our method can identify a single hub text that unreasonably achieves comparable or higher similarity scores than human-written reference captions in many images, thereby revealing the vulnerabilities in cross-modal encoders.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 29 of its 45 references that are in the map. First posted on arXiv 30 Apr 2026.
Builds on
29 of 45 references.
References · 29
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
- 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
- 2017Billion-scale similarity search with GPUscited by 2,497
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
- 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
- 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
- 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
- 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
17 more
- 2023In-Context Retrieval-Augmented Language Modelscited by 702
- 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 451
- 2018nocaps: novel object captioning at scalecited by 444
- 2023Text Embeddings Reveal (Almost) As Much As Textcited by 192
- 2022Truncation Sampling as Language Model Desmoothingcited by 123
- 2022AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilitiescited by 85
- 2023Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translationcited by 58
- 2014Improving zero-shot learning by mitigating the hubness problemcited by 54
- 2024RevPRAG: Revealing Poisoning Attacks in Retrieval-Augmented Generation through LLM Activation Analysiscited by 33
- 2025PR-Attack: Coordinated Prompt-RAG Attacks on Retrieval-Augmented Generation in Large Language Models via Bilevel Optimizationcited by 32
- 2023Balance Act: Mitigating Hubness in Cross-Modal Retrieval with Query and Gallery Bankscited by 20
- 2024Adversarial Hubness in Multi-Modal Retrievalcited by 9
- 2024I0T: Embedding Standardization Method Towards Zero Modality Gapcited by 8
- 2024Nearest Neighbor Normalization Improves Multimodal Retrievalcited by 7
- 2018Reconfigurable Inverted Indexcited by 5
- 2025Evaluation of Multilingual Image Captioning: How far can we get with CLIP models?cited by 5
- 2025Hacking Neural Evaluation Metrics with Single Hub Textcited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.