# Multimodal Generative Engine Optimization: Rank Manipulation for Vision-Language Model Rankers

arXiv: https://arxiv.org/abs/2601.12263 · HTML: https://mlsanity.com/paper/2601.12263/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 11 of its 15 references that are in the map.
- First posted on arXiv 18 Jan 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)

## Builds on (most cited first, 11 of 11)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Adversarial Search Engine Optimization for Large Language Models](https://mlsanity.com/paper/2406.18382/) (2024)
- [Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyond](https://mlsanity.com/paper/2403.10667/) (2024)
- [CheatAgent: Attacking LLM-Empowered Recommender Systems via LLM Agent](https://mlsanity.com/paper/2504.13192/) (2025)
- [Corpus Poisoning via Approximate Greedy Gradient Descent](https://mlsanity.com/paper/2406.05087/) (2024)
- [Stealthy Attack on Large Language Model based Recommendation](https://mlsanity.com/paper/2402.14836/) (2024)
- [Rec-GPT4V: Multimodal Recommendation with Large Vision-Language Models](https://mlsanity.com/paper/2402.08670/) (2024)
- [LLM Whisperer: An Inconspicuous Attack to Bias LLM Responses](https://mlsanity.com/paper/2406.04755/) (2024)
- [StealthRank: LLM Ranking Manipulation via Stealthy Prompt Optimization](https://mlsanity.com/paper/2504.05804/) (2025)
- [Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimization](https://mlsanity.com/paper/2510.06732/) (2025)
- [Dynamics of Adversarial Attacks on Large Language Model-Based Search Engines](https://mlsanity.com/paper/2501.00745/) (2025)

## Built on by (most cited first, 3 of 3)

- [PROGRESSLM: Towards Progress Reasoning in Vision-Language Models](https://mlsanity.com/paper/2601.15224/) (2026)
- [GEO-Bench: Benchmarking Ranking Manipulation in Generative Engine Optimization](https://mlsanity.com/paper/2605.29107/) (2026)
- [Do Vision Language Models Understand Human Engagement in Games?](https://mlsanity.com/paper/2603.18480/) (2026)

## Abstract (arXiv)

Vision-Language Models (VLMs) integrate visual and textual knowledge into unified representations that increasingly underpin modern retrieval and recommendation systems. However, it remains unclear how reliably these models utilize their cross-modal knowledge when ranking multimodal items, and whether their knowledge grounding can be subverted. In this paper, we expose a fundamental vulnerability in how VLMs apply multimodal knowledge for product ranking: through Multimodal Generative Engine Optimization (MGEO), we show that an adversary can manipulate a VLM's ranking decisions by jointly crafting imperceptible image perturbations and fluent textual suffixes that exploit the model's internal cross-modal knowledge coupling. Using an alternating optimization strategy, MGEO targets the deep interactions between visual and linguistic representations within the VLM, achieving rank manipulations that substantially exceed those of unimodal attacks and heuristic baselines powered by strong commercial models. Our findings reveal that surface-level content quality is insufficient for rank promotion; instead, direct alignment with the model's internal knowledge utilization mechanism is required. These results raise important questions on the faithfulness and robustness of knowledge grounding in multimodal foundation models, and motivate future work on defense mechanisms for multimodal retrieval systems. Code is available at: https://github.com/glad-lab/MGEO