mlsanityData to 14 Oct 2026

Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness

arXiv 2607.19322 ↗First posted 21 Jul 2026cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Rubric-based evaluation of open-ended generation faces a fundamental tension between expressiveness and reliability. Authoring a faithful rubric requires expressing the structure of the space of good answers: open-ended sets of acceptable options, ordered processes, and the relative importance of facts. Grading with the rubric requires a judge to score consistently, and judges are far more reliable on flat, binary checks than on rich structure. We resolve this tension with a two-level meta-rubric framework. A structured meta-rubric captures the grading criteria at authoring time, and fixed mechanical rules compile it into a flat checklist of binary, machine-gradable checks that an LLM judge scores reliably at evaluation time. We instantiate the framework as Gamut (Grounded Assessment of Multimodal Factuality), a benchmark for factual completeness in long-form generation. Gamut comprises 1,813 questions grounded in real wearable imagery across 10 diverse domains, each paired with an evidence-backed rubric verified by expert human annotators. Evaluating 14 frontier and open-weight models, we find Gamut genuinely challenging (best score 58.7% from Gemini 3.1 Pro), highly discriminative, and robust to the choice of judge.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 30 of its 50 references that are in the map. First posted on arXiv 21 Jul 2026.

Builds on

30 of 50 references.

References · 30

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  3. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  4. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  5. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
  6. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  7. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  8. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 251
  9. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 201
  10. 2024Long-form factuality in large language modelscited by 155
  11. 2024VERISCORE: Evaluating the factuality of verifiable claims in long-form text generationcited by 117
  12. 2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 101
18 more
  1. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 94
  2. 2025ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agentscited by 84
  3. 2024OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMscited by 72
  4. 2025The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Inputcited by 52
  5. 2025MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agentscited by 42
  6. 2024FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluationcited by 25
  7. 2025ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklistscited by 24
  8. 2026MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agentscited by 22
  9. 2024OLAPH: Improving Factuality in Biomedical Long-form Question Answeringcited by 20
  10. 2025The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factualitycited by 17
  11. 2025VeriFact: Enhancing Long-Form Factuality Evaluation with Refined Fact Extraction and Reference Factscited by 11
  12. 2025CRAG-MM: Multi-modal Multi-turn Comprehensive RAG Benchmarkcited by 10
  13. 2025FACTORY: A Challenging Human-Verified Prompt Set for Long-Form Factualitycited by 6
  14. 2025All Claims Are Equal, but Some Claims Are More Equal Than Others: Importance-Sensitive Factuality Evaluation of LLM Generationscited by 6
  15. 2026MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcomecited by 6
  16. 2025Long-Form Information Alignment Evaluation Beyond Atomic Factscited by 4
  17. 2026Beyond Precision: Importance-Aware Recall for Factuality Evaluation in Long-Form LLM Generationcited by 4
  18. 2025Chatbot Arena Meets Nuggets: Towards Explanations and Diagnostics in the Evaluation of LLM Responsescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026A Proposed Rubric for Evaluating Expressed Clinical Reasoning in Large Language Model Responsescited by 0
  2. 2026A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combinedcited by 0