mlsanityData to 14 Oct 2026

GrandJury: A Collaborative Machine Learning Model Evaluation Protocol for Dynamic Quality Rubrics

arXiv 2508.02926 ↗First posted 4 Aug 2025cs.LG cs.AI cs.HCcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Generative Machine Learning models have become central to modern systems, powering applications in creative writing, summarization, multi-hop reasoning, and context-aware dialogue. These models underpin large-scale AI assistants, workflow automation, and autonomous decision-making. In such domains, acceptable response is rarely absolute or static, but plural and highly context-dependent. Yet standard evaluation regimes still rely on static, benchmark-style tests, incentivizing optimization toward leaderboard scores rather than alignment with dynamic user needs or evolving realities. GrandJury introduces a formal evaluation protocol combining time-decayed aggregation, complete traceability, with the support of dynamic, transparent task rubric attribution, and multi-rater human judgment. Together, these elements enable pluralistic, accountable evaluation that captures evolving consensus and surfaces disagreement. We provide an open-source implementation (grandjury PyPI package) and a public collection of Large Language Model (LLM) inference outputs to illustrate the need and method. GrandJury provides a new paradigm for AI practitioners when evaluating machine learning outputs without absolute ground truth.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 15 of its 32 references that are in the map. First posted on arXiv 4 Aug 2025.

Builds on

15 of 32 references.

References · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  7. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  8. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  9. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  10. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  11. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  12. 2024Self-Preference Bias in LLM-as-a-Judgecited by 258
3 more
  1. 2018Troubling Trends in Machine Learning Scholarshipcited by 108
  2. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  3. 2024Automating the Enterprise with Foundation Modelscited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CollabEval: Enhancing LLM-as-a-Judge via Multi-Agent Collaboration
  2. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
  3. 2026EvalBuilder: Can LLMs Build Efficient Benchmarks?
  4. 2025Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation
  5. 2026GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation
  6. 2026QQJ: Quantifying Qualitative Judgment for Scalable and Human-Aligned Evaluation of Generative AI
  7. 2025When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMs
  8. 2024Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition