# A Black-Box Attack Model for Visually-Aware Recommender Systems

arXiv: https://arxiv.org/abs/2011.02701 · HTML: https://mlsanity.com/paper/2011.02701/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Nov 2020.

## Built on by (most cited first, 11 of 11)

- [Rank List Sensitivity of Recommender Systems to Interaction Perturbations](https://mlsanity.com/paper/2201.12686/) (2022)
- [DrunkAgent: Stealthy Memory Corruption in LLM-Powered Recommender Agents](https://mlsanity.com/paper/2503.23804/) (2025)
- [Shilling Black-box Review-based Recommender Systems through Fake Review Generation](https://mlsanity.com/paper/2306.16526/) (2023)
- [Semi-supervised Adversarial Learning for Complementary Item Recommendation](https://mlsanity.com/paper/2303.05812/) (2023)
- [Adversarial Item Promotion on Visually-Aware Recommender Systems by Guided Diffusion](https://mlsanity.com/paper/2312.15826/) (2023)
- [ToDA: Target-oriented Diffusion Attacker against Recommendation System](https://mlsanity.com/paper/2401.12578/) (2024)
- [Retrieval-Augmented Review Generation for Poisoning Recommender Systems](https://mlsanity.com/paper/2508.15252/) (2025)
- [Securing Visually-Aware Recommender Systems: An Adversarial Image Reconstruction and Detection Framework](https://mlsanity.com/paper/2306.07992/) (2023)
- [Adversarial Text Rewriting for Text-aware Recommender Systems](https://mlsanity.com/paper/2408.00312/) (2024)
- [Understanding the Effects of Adversarial Personalized Ranking Optimization Method on Recommendation Quality](https://mlsanity.com/paper/2107.13876/) (2021)
- [Securing Recommender System via Cooperative Training](https://mlsanity.com/paper/2401.12700/) (2024)

## Abstract (arXiv)

Due to the advances in deep learning, visually-aware recommender systems (RS) have recently attracted increased research interest. Such systems combine collaborative signals with images, usually represented as feature vectors outputted by pre-trained image models. Since item catalogs can be huge, recommendation service providers often rely on images that are supplied by the item providers. In this work, we show that relying on such external sources can make an RS vulnerable to attacks, where the goal of the attacker is to unfairly promote certain pushed items. Specifically, we demonstrate how a new visual attack model can effectively influence the item scores and rankings in a black-box approach, i.e., without knowing the parameters of the model. The main underlying idea is to systematically create small human-imperceptible perturbations of the pushed item image and to devise appropriate gradient approximation methods to incrementally raise the pushed item's score. Experimental evaluations on two datasets show that the novel attack model is effective even when the contribution of the visual features to the overall performance of the recommender system is modest.