OenoBench: A Wine-Domain Benchmark for Knowledge-Grounded Evaluation of Large Language Models
Not on the map. Why?
Abstract (arXiv)
We introduce OenoBench, a wine-domain knowledge benchmark of 3,266 multiple-choice questions across six pillars (regions, grape varieties, viticulture, winemaking, producers, business) and four difficulty tiers. The corpus is built from 38,104 atomic, source-anchored facts extracted by 35 provenance-verified scrapers from government registries (INAO, TTB, OIV), peer-reviewed journals, and Wikipedia/Wikidata. Our methodological contribution is an LLM-driven pipeline in which language models reformat verified facts and audit the result, but never serve as the source of truth: every claim traces to a URL, every question is generated by one of five strategies across five generator families, and every question is scored by a nine-agent audit calibrated against a human gold sheet via Cohen's kappa. Evaluating sixteen frontier configurations, we find: (i) overall accuracy spans 53%-84%, led by o3 at 83.6%; (ii) reasoning-mode lift concentrates in DeepSeek R1 (+6.8pp) and is absent in Claude Opus and Gemini Pro; (iii) Anthropic shows +9pp self preference on its own questions while Google shows -8pp inverse preference; (iv) frontier open-weight models share the cost-vs-accuracy Pareto frontier with proprietary reasoning models; and (v) every config gains around 33pp on closed-book solvable items, revealing a parametric-recall ceiling that only the contextual slice avoids. We release corpus, audit findings, human-review app, and construction code under CC-BY-SA-4.0.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 14 of its 19 references that are in the map. First posted on arXiv 20 Aug 2026.
Builds on
14 of 19 references.
References · 14
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
- 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
- 2018Datasheets for Datasetscited by 1,371
- 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
- 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
- 2023Towards Understanding Sycophancy in Language Modelscited by 988
- 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
- 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
- 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
- 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 328
2 more
- 2022Data Contamination: From Memorization to Exploitationcited by 234
- 2024Croissant: A Metadata Format for ML-Ready Datasetscited by 79
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements
- 2025OKBench: Democratizing LLM Evaluation with Fully Automated, On-Demand, Open Knowledge Benchmarking
- 2026Beyond Questions: Evaluating LLM's Knowledge Expression
- 2025Follow the Path: Reasoning over Knowledge Graph Paths to Improve Large Language Model Factuality
- 2023DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text
- 2024metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models
- 2025Automated MCQA Benchmarking at Scale: Evaluating Reasoning Traces as Retrieval Sources for Domain Adaptation of Small Language Models
- 2026Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities