mlsanityData to 14 Oct 2026

RADAR: Rubric-Aware Dependency and Redundancy Analysis for LLM-as-Judge Evaluation

arXiv 2608.01810 ↗First posted 3 Aug 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Rubric-based LLM-as-judge pipelines often assume that evaluation criteria provide independent signals. In practice, however, criteria can be behaviorally coupled: improving one criterion may systematically change scores on another, distorting aggregate scores used in model-release or product-update decisions. We introduce RADAR, a lightweight preflight diagnostic framework for estimating such coupling before large-scale evaluation. Given a rubric, RADAR generates targeted synthetic probes, scores each probe on all criteria, and produces a directional coupling matrix that shows which criteria co-score and how. We validate RADAR on three industry-relevant evaluation settings: NVIDIA HelpSteer2, SumPubMed, and the Yale-Salesforce SummEval benchmark. Using only a small number of probes per criterion, RADAR recovers human inter-criterion correlation structure (Pearson r > 0.84) and provides practitioners with concrete audit signals about redundancy, hierarchy, and aggregation sensitivity before committing to large-scale judging.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 28 references that are in the map. First posted on arXiv 3 Aug 2026.

Builds on

19 of 28 references.

References · 19

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  4. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  5. 2024Self-Preference Bias in LLM-as-a-Judgecited by 258
  6. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
  7. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 201
  8. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 153
  9. 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 144
  10. 2025JudgeLRM: Large Reasoning Models as a Judgecited by 78
  11. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 62
  12. 2024Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarkingcited by 35
7 more
  1. 2024Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructionscited by 34
  2. 2024A Comprehensive Evaluation of Cognitive Biases in LLMscited by 34
  3. 2024HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decompositioncited by 33
  4. 2026Learning to Judge: LLMs Designing and Applying Evaluation Rubricscited by 19
  5. 2025When Judgment Becomes Noise: How Design Failures in LLM Judge Benchmarks Silently Undermine Validitycited by 11
  6. 2026RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generationcited by 11
  7. 2025Through the Judge's Eyes: Inferred Thinking Traces Improve Reliability of LLM Raterscited by 2

Built on by

Not cited yet.