# Retrieve, Annotate, Evaluate, Repeat: Leveraging Multimodal LLMs for Large-Scale Product Retrieval Evaluation

arXiv: https://arxiv.org/abs/2409.11860 · HTML: https://mlsanity.com/paper/2409.11860/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 6 of its 26 references that are in the map.
- First posted on arXiv 18 Sep 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)

## Builds on (most cited first, 6 of 6)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [Large language models can accurately predict searcher preferences](https://mlsanity.com/paper/2309.10621/) (2023)
- [Synthetic Test Collections for Retrieval Evaluation](https://mlsanity.com/paper/2405.07767/) (2024)
- [LLMs Can Patch Up Missing Relevance Judgments in Evaluation](https://mlsanity.com/paper/2405.04727/) (2024)

## Built on by (most cited first, 12 of 12)

- [Automatic Dataset Generation for Knowledge Intensive Question Answering Tasks](https://mlsanity.com/paper/2505.14212/) (2025)
- [CliniQ: A Multi-faceted Benchmark for Electronic Health Record Retrieval with Semantic Match Assessment](https://mlsanity.com/paper/2502.06252/) (2025)
- [AgentDR: Dynamic Recommendation with Implicit Item-Item Relations via LLM-based Agents](https://mlsanity.com/paper/2510.05598/) (2025)
- [Agentic Personalized Fashion Recommendation in the Age of Generative AI: Challenges, Opportunities, and Evaluation](https://mlsanity.com/paper/2508.02342/) (2025)
- [Image-Seeking Intent Prediction for Cross-Device Product Search](https://mlsanity.com/paper/2511.14764/) (2025)
- [Personagram: Bridging Personas and Product Design for Creative Ideation with Multimodal LLMs](https://mlsanity.com/paper/2602.06197/) (2026)
- [Scaling Search Relevance: Augmenting App Store Ranking with LLM-Generated Judgments](https://mlsanity.com/paper/2602.23234/) (2026)
- [A Modular LLM Framework for Explainable Price Outlier Detection](https://mlsanity.com/paper/2603.20636/) (2026)
- [Deepchecks: Evaluating Retrieval-Augmented Generation (RAG)](https://mlsanity.com/paper/2605.14488/) (2026)
- [VCG: A Multimodal Retrieval Framework for E-Commerce Video Feeds under Extreme Cold-Start Conditions](https://mlsanity.com/paper/2606.19627/) (2026)
- [HIERA: Hierarchical Multi-Agent Relevance Assessment for Content Discovery Systems](https://mlsanity.com/paper/2608.00785/) (2026)
- [On Evaluating and Improving Conversational Agents in Production](https://mlsanity.com/paper/2609.32092/) (2026)

## Abstract (arXiv)

Evaluating production-level retrieval systems at scale is a crucial yet challenging task due to the limited availability of a large pool of well-trained human annotators. Large Language Models (LLMs) have the potential to address this scaling issue and offer a viable alternative to humans for the bulk of annotation tasks. In this paper, we propose a framework for assessing the product search engines in a large-scale e-commerce setting, leveraging Multimodal LLMs for (i) generating tailored annotation guidelines for individual queries, and (ii) conducting the subsequent annotation task. Our method, validated through deployment on a large e-commerce platform, demonstrates comparable quality to human annotations, significantly reduces time and cost, facilitates rapid problem discovery, and provides an effective solution for production-level quality control at scale.