mlsanityData to 14 Oct 2026

How to Build a Benchmark That Actually Measures Taste

arXiv 2610.14707 ↗First posted 10 Oct 2026cs.AI cs.CL cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Expert-aligned "taste" benchmarks should reward craft judgment, not the ability to game the test. We introduce TasteBench, a pairwise good-vs-bad benchmark across visual, brand, book, and tech items under a frozen rubric with expert-only gold (panel n>=3; percent agreement >=0.60 primary gate; weak engagement labels train/analysis-only), together with a control protocol that separates craft judgment from test-gaming. Applying the protocol to a Round-1 pilot (primary_human_expert; three biological human experts; n=250 pairs) surfaces and quantifies three validity threats: two construction artifacts that inflate the trivial-baseline layer, and a difficulty ceiling that inflates every score. (1) Position: the preferred item is slot A with probability 0.956, so an always_item_a policy scores 0.956 (fair chance 0.5). (2) Lexical leakage: a filename token cue scores 0.912. (3) Difficulty ceiling: craft domains show near-ceiling human agreement (book/brand/tech 1.0), and even under A/B randomization (eval_assignment_seed7; gold_side_A ~0.47) all five VLM cards reach 1.0 on craft, with only visual separating (~0.84-0.91; a post-hoc hard subset, n=107, leaves craft at 1.0). The protocol (expert-only gold, a weak-label firewall, side randomization at construction, filename scrubbing, and hard filters) makes each artifact measurable and, for the two construction artifacts, correctable; we release a reference eval harness and a side-randomization tool, and show the tool removes the position skew (0.916 to 0.495/0.505). The result is a calibrated view of what the benchmark does and does not yet measure: position and leakage are fixable, whereas the craft-domain ceiling is a pair-difficulty limit that Round-2 must address. We do not evaluate engagement, era/circle conditioning, or calibration here, and we claim no general taste SOTA.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 9 of its 22 references that are in the map. First posted on arXiv 10 Oct 2026.

Builds on

9 of 22 references.

References · 9

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2017On Calibration of Modern Neural Networkscited by 4,960
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  7. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  8. 2016Unbiased Learning-to-Rank with Biased Feedbackcited by 157
  9. 2016Photo Aesthetics Ranking Network with Attributes and Content Adaptationcited by 125

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks
  2. 2026Auditing LLM Benchmarks with Item Response Theory
  3. 2025Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcuts
  4. 2026FoodSense: A Multisensory Food Dataset and Benchmark for Predicting Taste, Smell, Texture, and Sound from Images
  5. 2026A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks
  6. 2026SESSE: Sketch, Expand, Sort, Summarize, Evaluate -- LLM-as-Judge Evaluation via Structured Decomposition
  7. 2026Visual Aesthetic Benchmark: Can Frontier Models Judge Beauty?
  8. 2026VLM Judges Can Rank but Cannot Score: Task-Dependent Uncertainty in Multimodal Evaluation