# Self-rationalization improves LLM as a fine-grained judge

arXiv: https://arxiv.org/abs/2410.05495 · HTML: https://mlsanity.com/paper/2410.05495/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 36 references that are in the map.
- First posted on arXiv 7 Oct 2024.

## Builds on (most cited first, 17 of 17)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Self-Rewarding Language Models](https://mlsanity.com/paper/2401.10020/) (2024)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [Explain Yourself! Leveraging Language Models for Commonsense Reasoning](https://mlsanity.com/paper/1906.02361/) (2019)
- [PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization](https://mlsanity.com/paper/2306.05087/) (2023)
- [Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations](https://mlsanity.com/paper/1703.03717/) (2017)
- [Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge](https://mlsanity.com/paper/2407.19594/) (2024)
- [Benchmarking Foundation Models with Language-Model-as-an-Examiner](https://mlsanity.com/paper/2306.04181/) (2023)
- [BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling](https://mlsanity.com/paper/2406.00832/) (2024)
- [Branch-Solve-Merge Improves Large Language Model Evaluation and Generation](https://mlsanity.com/paper/2310.15123/) (2023)
- [The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation](https://mlsanity.com/paper/2308.07286/) (2023)
- [The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models](https://mlsanity.com/paper/2406.05761/) (2024)
- [Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraint](https://mlsanity.com/paper/2401.06081/) (2024)
- [West-of-N: Synthetic Preferences for Self-Improving Reward Models](https://mlsanity.com/paper/2401.12086/) (2024)
- [HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedback](https://mlsanity.com/paper/2403.08309/) (2024)

## Built on by (most cited first, 27 of 27)

- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems](https://mlsanity.com/paper/2504.09037/) (2025)
- [TRAIL: Trace Reasoning and Agentic Issue Localization](https://mlsanity.com/paper/2505.08638/) (2025)
- [Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge](https://mlsanity.com/paper/2501.18099/) (2025)
- [J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning](https://mlsanity.com/paper/2505.10320/) (2025)
- [BED-LLM: Intelligent Information Gathering with LLMs and Bayesian Experimental Design](https://mlsanity.com/paper/2508.21184/) (2025)
- [TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them](https://mlsanity.com/paper/2509.21117/) (2025)
- [Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judge](https://mlsanity.com/paper/2502.12501/) (2025)
- [Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments](https://mlsanity.com/paper/2504.17087/) (2025)
- [No Need for Explanations: LLMs can implicitly learn from mistakes in-context](https://mlsanity.com/paper/2502.08550/) (2025)
- [RLAC: Reinforcement Learning with Adversarial Critic for Free-Form Generation Tasks](https://mlsanity.com/paper/2511.01758/) (2025)
- [Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplifications and Resistance in Multi-Agent Based LLM-as-Judge](https://mlsanity.com/paper/2505.19477/) (2025)
- [Quantitative LLM Judges](https://mlsanity.com/paper/2506.02945/) (2025)
- [LLM-as-Judge on a Budget](https://mlsanity.com/paper/2602.15481/) (2026)
- [Personalized Reward Modeling for Text-to-Image Generation](https://mlsanity.com/paper/2511.19458/) (2025)
- [Reasoning Path and Latent State Analysis for Multi-view Visual Spatial Reasoning: A Cognitive Science Perspective](https://mlsanity.com/paper/2512.02340/) (2025)
- [Meta-Judging with Large Language Models: Concepts, Methods, and Challenges](https://mlsanity.com/paper/2601.17312/) (2026)
- [Cross-Lingual LLM-Judge Transfer via Evaluation Decomposition](https://mlsanity.com/paper/2603.18557/) (2026)
- [When Languages Disagree: Self-Evolving Multilingual LLM Judges](https://mlsanity.com/paper/2606.08092/) (2026)
- [References Improve LLM Alignment in Non-Verifiable Domains](https://mlsanity.com/paper/2602.16802/) (2026)
- [Fast-Slow Thinking RM: Efficient Integration of Scalar and Generative Reward Models](https://mlsanity.com/paper/2603.20212/) (2026)
- [Learned-Rule-Augmented Large Language Model Evaluators](https://mlsanity.com/paper/2512.01958/) (2025)
- [Exploring the features used for summary evaluation by Human and GPT](https://mlsanity.com/paper/2512.19620/) (2025)
- [Robust LLM Performance Certification via Constrained Maximum Likelihood Estimation](https://mlsanity.com/paper/2604.03257/) (2026)
- [Macro: Enhancing Multilingual Counterfactual Explanations through Alignment-as-Preference Optimization](https://mlsanity.com/paper/2605.11632/) (2026)
- [AwarenessBench: Assessing Cognitive Capabilities of Language Models](https://mlsanity.com/paper/2609.35409/) (2026)
- [JudgeMoE: Distributional Aggregation for LLM-as-a-Judge](https://mlsanity.com/paper/2610.07109/) (2026)

## Abstract (arXiv)

LLM-as-a-judge models have been used for evaluating both human and AI generated content, specifically by providing scores and rationales. Rationales, in addition to increasing transparency, help models learn to calibrate its judgments. Enhancing a model's rationale can therefore improve its calibration abilities and ultimately the ability to score content. We introduce Self-Rationalization, an iterative process of improving the rationales for the judge models, which consequently improves the score for fine-grained customizable scoring criteria (i.e., likert-scale scoring with arbitrary evaluation criteria). Self-rationalization works by having the model generate multiple judgments with rationales for the same input, curating a preference pair dataset from its own judgements, and iteratively fine-tuning the judge via DPO. Intuitively, this approach allows the judge model to self-improve by learning from its own rationales, leading to better alignment and evaluation accuracy. After just two iterations -- while only relying on examples in the training set -- human evaluation shows that our judge model learns to produce higher quality rationales, with a win rate of 62% on average compared to models just trained via SFT on rationale . This judge model also achieves high scoring accuracy on BigGen Bench and Reward Bench, outperforming even bigger sized models trained using SFT with rationale, self-consistency or best-of-N sampling by 3% to 9%.