# A Comparative Evaluation of Embeddings and LLMs in a Greek Book Publisher Setting - The CUP Dataset

arXiv: https://arxiv.org/abs/2607.21274 · HTML: https://mlsanity.com/paper/2607.21274/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 2 of its 18 references that are in the map.
- First posted on arXiv 23 Jul 2026.

## Builds on (most cited first, 2 of 2)

- [Training Sparse Mixture Of Experts Text Embedding Models](https://mlsanity.com/paper/2502.07972/) (2025)
- [GreekBarBench: A Challenging Benchmark for Free-Text Legal Reasoning and Citations](https://mlsanity.com/paper/2505.17267/) (2025)

## Abstract (arXiv)

We present CUP, a Greek book retrieval benchmark consisting of 868 catalog records and 104 expert-annotated queries with graded relevance judgments. We evaluate sparse (BM25), dense (sentence-transformers), hybrid, and LLM-assisted retrieval methods in this book-search setting. Multilingual embeddings outperform Greek-specific models, while hybrid retrieval performs best overall. A query-level analysis shows that BM25 excels at named-entity queries, while dense and hybrid methods improve natural-language, noisy, cross-lingual, and concept queries. Field-aware prompting has model-specific effects, while LLM TOC summarization improves TOC-only retrieval and LLM post-filtering improves early-stage retrieval at a high cost. Overall, CUP enables real-world evaluation of Greek retrieval across lexical, semantic, noisy, and cross-lingual queries.