mlsanityData to 14 Oct 2026

Think-J: Learning to Think for Generative LLM-as-a-Judge

arXiv 2505.14268 ↗First posted 20 May 2025cs.CL cs.AIcited by 21

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

LLM-as-a-Judge refers to the automatic modeling of preferences for responses generated by Large Language Models (LLMs), which is of significant importance for both LLM evaluation and reward modeling. Although generative LLMs have made substantial progress in various tasks, their performance as LLM-Judge still falls short of expectations. In this work, we propose Think-J, which improves generative LLM-as-a-Judge by learning how to think. We first utilized a small amount of curated data to develop the model with initial judgment thinking capabilities. Subsequently, we optimize the judgment thinking traces based on reinforcement learning (RL). We propose two methods for judgment thinking optimization, based on offline and online RL, respectively. The offline method requires training a critic model to construct positive and negative examples for learning. The online method defines rule-based reward as feedback for optimization. Experimental results showed that our approach can significantly enhance the evaluation capability of generative LLM-Judge, surpassing both generative and classifier-based LLM-Judge without requiring extra human annotations.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 47 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

31 of 47 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025RM-R1: Reward Modeling as Reasoning5 May
  2. 2025Inference-Time Scaling for Generalist Reward Modeling3 Apr
  3. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternatives7 Nov
  4. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 25

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  8. 2024A Survey on LLM-as-a-Judgecited by 1,423
  9. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  10. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  11. 2025LIMO: Less is More for Reasoningcited by 498
  12. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
13 more
  1. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  2. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 176
  3. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 171
  4. 2023Generative Judge for Evaluating Alignmentcited by 169
  5. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 130
  6. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 94
  7. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  8. 2025Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuningcited by 90
  9. 2025JudgeLRM: Large Reasoning Models as a Judgecited by 78
  10. 2023CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generationcited by 66
  11. 2024Improving Reward Models with Synthetic Critiquescited by 44
  12. 2024CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolutioncited by 42
  13. 2025AIR: Complex Instruction Generation via Automatic Iterative Refinementcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 14.

21 papers.

Sort:

By area:

  1. 2025CriticLean: Critic-Guided Reinforcement Learning for Mathematical Formalizationcited by 16
  2. 2025ReLook: Vision-Grounded RL with a Multimodal LLM Critic for Agentic Web Codingcited by 14
  3. 2026Reward Modeling from Natural Language Human Feedbackcited by 11
  4. 2026Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimizationcited by 9
  5. 2025Trust but Verify! A Survey on Verification Design for Test-time Scalingcited by 8
  6. 2025SCI-Verifier: Scientific Verifier with Thinkingcited by 6
  7. 2025S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Modelscited by 5
  8. 2025Rubric-Conditioned LLM Grading: Alignment, Uncertainty, and Robustnesscited by 5
  9. 2026Aletheia: What Makes RLVR For Code Verifiers Tick?cited by 5
  10. 2026DeepSurvey-Bench: Evaluating Academic Value of Automatically Generated Scientific Surveyscited by 4
  11. 2026CAMEL: Confidence-Gated Reflection for Reward Modelingcited by 3
  12. 2026RM-Distiller: Exploiting Generative LLM for Reward Model Distillationcited by 2
9 more
  1. 2026CE-RM: A Pointwise Generative Reward Model Optimized via Two-Stage Rollout and Unified Criteriacited by 2
  2. 2026Long-form RewardBench: Evaluating Reward Models for Long-form Generationcited by 2
  3. 2026RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generatorcited by 1
  4. 2025The Answer Lies Within: Self-Derived Rewards Enable Explainable Relation Extractioncited by 0
  5. 2026Towards Spec Learning: Inference-Time Alignment from Preference Pairscited by 0
  6. 2026DynaContext: Self-Improving Dynamic Contextualization of Optimized Prompts for Heterogeneous Parameter Extractioncited by 0
  7. 2026Beyond Correctness: Validity-Oriented Evaluation of Biomedical LLM Judgescited by 0
  8. 2026JudgePanel: A Compact Judge with Panel Deliberation via Adaptive Multi-Reward Reinforcement Learningcited by 0
  9. 2026LegalRewardBench: Benchmarking Reward Models for Legal Preference Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Improve LLM-as-a-Judge Ability as a General Ability
  2. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning
  3. 2024Beyond Scalar Reward Model: Learning Generative Judge from Preference Data
  4. 2024Direct Judgement Preference Optimization
  5. 2025J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization
  6. 2025Explicit Reasoning Makes Better Judges: A Systematic Study on Accuracy, Efficiency, and Robustness
  7. 2025Improving LLM-as-a-Judge Inference with the Judgment Distribution
  8. 2024Generative Reward Models