mlsanityData to 7 Oct 2026

Advancing Relevance Measurement with Vision-Language Models for Web-Scale Search

arXiv 2608.02446 ↗First posted 3 Aug 2026cs.IR cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Relevance evaluation plays a crucial role in personalized search systems, serving as a guardrail alongside user engagement metrics to ensure that search results align with user queries and intent. While human annotation is the traditional method for relevance evaluation, its high cost and long turnaround time limit its scalability. In this work, we present a VLM-based automated relevance evaluation pipeline deployed within Pinterest Search for online A/B experiments. We rigorously validate the alignment between VLM-generated judgments and human annotations, demonstrating that VLMs can provide reliable relevance measurement for experiments while greatly improving the evaluation efficiency. Leveraging VLM-based labeling further unlocks opportunities to expand the query set, optimize sampling design, and efficiently assess a wider range of search experiences at scale. This approach leads to higher-quality relevance metrics and significantly reduces the Minimum Detectable Effects (MDEs) in online experiment measurements.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 13 of its 30 references that are in the map. First posted on arXiv 3 Aug 2026.

Builds on

13 of 30 references.

References · 13

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2024A Survey on LLM-as-a-Judgecited by 1,384
  5. 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 142
  6. 2023Large language models can accurately predict searcher preferencescited by 129
  7. 2023Perspectives on Large Language Models for Relevance Judgmentcited by 84
  8. 2024UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessorcited by 41
  9. 2023One-Shot Labeling for Automatic Relevance Estimationcited by 31
  10. 2024Query Performance Prediction using Relevance Judgments Generated by Large Language Modelscited by 18
  11. 2024Can We Use Large Language Models to Fill Relevance Judgment Holes?cited by 17
  12. 2025Benchmarking LLM-based Relevance Judgment Methodscited by 14
1 more
  1. 2024Toward Automatic Relevance Judgment using Vision--Language Models for Image--Text Retrieval Evaluationcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026PRECISE: Reducing the Bias of LLM Evaluations Using Prediction-Powered Ranking Estimation
  2. 2025Automated Query-Product Relevance Labeling using Large Language Models for E-commerce Search
  3. 2025LORE: A Large Generative Model for Search Relevance
  4. 2025RAIR: A Rule-Aware Benchmark Uniting Challenging Long-Tail and Visual Salience Subset for E-commerce Relevance Assessment
  5. 2024A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look
  6. 2025A Human-AI Comparative Analysis of Prompt Sensitivity in LLM-Based Relevance Judgment
  7. 2026Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarks
  8. 2024Evaluating Cost-Accuracy Trade-offs in Multimodal Search Relevance Judgements