mlsanityData to 14 Oct 2026

Towards a Reliable and Practical Eval Pipeline

arXiv 2609.00805 ↗First posted 1 Sep 2026cs.AI cs.SEcited by 0

Not on the map. Why?

Abstract (arXiv)

LLM-based software systems increasingly require effective "evals" as quality gates in the development lifecycle. However, existing work typically addresses individual aspects of eval reliability rather than the full set of practical requirements. We present an end-to-end eval pipeline that combines eval checklist creation, with learned aggregation for checklist responses, to improve agreement across LLM judges and accuracy against human judgments. The framework additionally pro- vides self-consistency, explanations, and prediction uncertainty, and we empirically demonstrate its effectiveness.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 27 references that are in the map. First posted on arXiv 1 Sep 2026.

Builds on

16 of 27 references.

References · 16

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  4. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  5. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  6. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  7. 2016Distribution-Free Predictive Inference For Regressioncited by 668
  8. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  9. 2025Reasoning Models Don't Always Say What They Thinkcited by 415
  10. 2018Consistent Individualized Feature Attribution for Tree Ensemblescited by 285
  11. 2024Are self-explanations from Large Language Models faithful?cited by 101
  12. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 87
4 more
  1. 2024HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decompositioncited by 33
  2. 2024DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregationcited by 12
  3. 2024Can LLMs be Good Graph Judge for Knowledge Graph Construction?cited by 11
  4. 2025VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captionscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliability
  2. 2025From Code to Courtroom: LLMs as the New Software Judges
  3. 2026Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering
  4. 2025LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead ↗
  5. 2026One-Eval: An Agentic System for Automated and Traceable LLM Evaluation
  6. 2026AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judge
  7. 2026TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenance
  8. 2026Autorubric: A Unifying Framework for Rubric-Based LLM Evaluation on Non-Verifiable Tasks