# Turning Adversaries into Allies: Reversing Typographic Attacks for Multimodal E-Commerce Product Retrieval

arXiv: https://arxiv.org/abs/2511.05325 · HTML: https://mlsanity.com/paper/2511.05325/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 30 references that are in the map.
- First posted on arXiv 7 Nov 2025.

## Builds on (most cited first, 16 of 16)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [CLIPPO: Image-and-Language Understanding from Pixels Only](https://mlsanity.com/paper/2212.08045/) (2022)
- [Vision-LLMs Can Fool Themselves with Self-Generated Typographic Attacks](https://mlsanity.com/paper/2402.00626/) (2024)
- [A Sober Look at the Robustness of CLIPs to Spurious Features](https://mlsanity.com/paper/2403.11497/) (2024)
- [LeGrad: An Explainability Method for Vision Transformers via Feature Formation Sensitivity](https://mlsanity.com/paper/2404.03214/) (2024)
- [Reading Isn't Believing: Adversarial Attacks On Multi-Modal Neurons](https://mlsanity.com/paper/2103.10480/) (2021)
- [CommerceMM: Large-Scale Commerce MultiModal Representation Learning with Omni Retrieval](https://mlsanity.com/paper/2202.07247/) (2022)
- [Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filters](https://mlsanity.com/paper/2403.02677/) (2024)
- [Parrot Captions Teach CLIP to Spot Text](https://mlsanity.com/paper/2312.14232/) (2023)
- [Exploring Typographic Visual Prompts Injection Threats in Cross-Modality Generation Models](https://mlsanity.com/paper/2503.11519/) (2025)
- [Language-biased image classification: evaluation based on semantic representations](https://mlsanity.com/paper/2201.11014/) (2022)
- [Delving into E-Commerce Product Retrieval with Vision-Language Pre-training](https://mlsanity.com/paper/2304.04377/) (2023)
- [End-to-end multi-modal product matching in fashion e-commerce](https://mlsanity.com/paper/2403.11593/) (2024)
- [Multimodal semantic retrieval for product search](https://mlsanity.com/paper/2501.07365/) (2025)

## Abstract (arXiv)

Multimodal product retrieval systems in e-commerce platforms rely on effectively combining visual and textual signals to improve search relevance and user experience. However, vision-language models such as CLIP are vulnerable to typographic attacks, where misleading or irrelevant text embedded in images skews model predictions. In this work, we propose a novel method that reverses the logic of typographic attacks by rendering relevant textual content (e.g., titles, descriptions) directly onto product images to perform vision-text compression, thereby strengthening image-text alignment and boosting multimodal product retrieval performance. We evaluate our method on three vertical-specific e-commerce datasets (sneakers, handbags, and trading cards) using six state-of-the-art vision foundation models. Our experiments demonstrate consistent improvements in unimodal and multimodal retrieval accuracy across categories and model families. Our findings suggest that visually rendering product metadata is a simple yet effective enhancement for zero-shot multimodal retrieval in e-commerce applications.