# Semantic Search Evaluation

arXiv: https://arxiv.org/abs/2410.21549 · HTML: https://mlsanity.com/paper/2410.21549/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 3 of its 17 references that are in the map.
- First posted on arXiv 28 Oct 2024.

## Builds on (most cited first, 3 of 3)

- [On Application of Learning to Rank for E-Commerce Search](https://mlsanity.com/paper/1903.04263/) (2019)
- [Report on the 1st Workshop on Large Language Model for Evaluation in Information Retrieval (LLM4Eval 2024) at SIGIR 2024](https://mlsanity.com/paper/2408.05388/) (2024)
- [LARR: Large Language Model Aided Real-time Scene Recommendation with Semantic Understanding](https://mlsanity.com/paper/2408.11523/) (2024)

## Abstract (arXiv)

We propose a novel method for evaluating the performance of a content search system that measures the semantic match between a query and the results returned by the search system. We introduce a metric called "on-topic rate" to measure the percentage of results that are relevant to the query. To achieve this, we design a pipeline that defines a golden query set, retrieves the top K results for each query, and sends calls to GPT 3.5 with formulated prompts. Our semantic evaluation pipeline helps identify common failure patterns and goals against the metric for relevance improvements.