# SedarEval: Automated Evaluation using Self-Adaptive Rubrics

arXiv: https://arxiv.org/abs/2501.15595 · HTML: https://mlsanity.com/paper/2501.15595/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 12 references that are in the map.
- First posted on arXiv 26 Jan 2025.

## Builds on (most cited first, 5 of 5)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [Hurdles to Progress in Long-form Question Answering](https://mlsanity.com/paper/2103.06332/) (2021)
- [A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods](https://mlsanity.com/paper/2104.00054/) (2021)

## Built on by (most cited first, 19 of 19)

- [Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning](https://mlsanity.com/paper/2508.16949/) (2025)
- [Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey](https://mlsanity.com/paper/2503.22458/) (2025)
- [Rubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubrics](https://mlsanity.com/paper/2503.23989/) (2025)
- [Mathematical Proof as a Litmus Test: Revealing Failure Modes of Advanced Large Reasoning Models](https://mlsanity.com/paper/2506.17114/) (2025)
- [RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation](https://mlsanity.com/paper/2603.20882/) (2026)
- [Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs](https://mlsanity.com/paper/2506.10054/) (2025)
- [ShoppingComp: Are LLMs Really Ready for Your Shopping Cart?](https://mlsanity.com/paper/2511.22978/) (2025)
- [Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges](https://mlsanity.com/paper/2602.13576/) (2026)
- [Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judge](https://mlsanity.com/paper/2603.11027/) (2026)
- [From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape](https://mlsanity.com/paper/2606.08625/) (2026)
- [CultureCLIP: Empowering CLIP with Cultural Awareness through Synthetic Images and Contextualized Captions](https://mlsanity.com/paper/2507.06210/) (2025)
- [LLM-as-Judge in Education: A Curriculum-Grounded Marking Pipeline](https://mlsanity.com/paper/2606.17507/) (2026)
- [REVEAL: A Rubric-Guided Agent for Explicit Evidence Sufficiency Verificationin Long-Video Question Answering](https://mlsanity.com/paper/2608.08612/) (2026)
- [Paper Reconstruction Evaluation: Evaluating Presentation and Hallucination in AI-written Papers](https://mlsanity.com/paper/2604.01128/) (2026)
- [Feedback-to-Rubrics: Can We Learn Expert Criteria from Inline Comments?](https://mlsanity.com/paper/2605.29857/) (2026)
- [Rubric-as-Experts: Case-Specific MQM Rubrics for Translation Error Span Detection](https://mlsanity.com/paper/2606.21559/) (2026)
- [Agentic systems for breast cancer treatment recommendations](https://mlsanity.com/paper/2607.12051/) (2026)
- [Beyond Correctness: Benchmarking and Aligning Response Behaviors in Hybrid-Thinking MLLMs](https://mlsanity.com/paper/2608.12781/) (2026)
- [Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents](https://mlsanity.com/paper/2608.31076/) (2026)

## Abstract (arXiv)

The evaluation paradigm of LLM-as-judge gains popularity due to its significant reduction in human labor and time costs. This approach utilizes one or more large language models (LLMs) to assess the quality of outputs from other LLMs. However, existing methods rely on generic scoring rubrics that fail to consider the specificities of each question and its problem-solving process, compromising precision and stability in assessments. Inspired by human examination scoring processes, we propose a new evaluation paradigm based on self-adaptive rubrics. Specifically, we create detailed scoring rubrics for each question, capturing the primary and secondary criteria in a structured format of scoring and deduction points that mimic a human evaluator's analytical process. Building on this paradigm, we further develop a novel benchmark called SedarEval, which covers a range of domains including long-tail knowledge, mathematics, coding, and logical reasoning. SedarEval consists of 1,000 meticulously crafted questions, each with its own self-adaptive rubric. To further streamline the evaluation, we train a specialized evaluator language model (evaluator LM) to supplant human graders. Using the same training data, our evaluator LM achieves a higher concordance rate with human grading results than other paradigms, including GPT-4, highlighting the superiority and efficiency of our approach. We release our dataset at https://github.com/wwn1233/sedareval.