mlsanityData to 7 Oct 2026

LLM-as-a-Judge & Reward Model: What They Can and Cannot Do

arXiv 2409.11239 ↗First posted 17 Sep 2024cs.CLcited by 32

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

LLM-as-a-Judge and reward models are widely used alternatives of multiple-choice questions or human annotators for large language model (LLM) evaluation. Their efficacy shines in evaluating long-form responses, serving a critical role as evaluators of leaderboards and as proxies to align LLMs via reinforcement learning. However, despite their popularity, their effectiveness in diverse contexts, such as non-English prompts, factual verification, or challenging questions, remains unexplored. In this paper, we conduct a comprehensive analysis of automated evaluators, reporting several key findings on their behavior. First, we discover that English evaluation capabilities significantly influence language-specific evaluation capabilities, often more than the language proficiency itself, enabling evaluators trained in English to easily transfer their skills to other languages. Second, we identify critical shortcomings, where LLMs fail to detect and penalize errors, such as factual inaccuracies, cultural misrepresentations, and the presence of unwanted language. Finally, we find that state-of-the-art evaluators struggle with challenging prompts, in either English or Korean, underscoring their limitations in assessing or generating complex reasoning questions. We release the dataset and codes used.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 46 of its 72 references that are in the map. First posted on arXiv 17 Sep 2024.

Builds on

46 of 72 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators25 Mar
  2. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference7 Mar
  3. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Models12 Oct
  4. 2023Pretraining Language Models with Human Preferences16 Feb

Other references · 42

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  8. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  9. 2024Qwen2 Technical Reportcited by 2,076
  10. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  11. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  12. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
30 more
  1. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  2. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  3. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  4. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  5. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  6. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  7. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  8. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  9. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  10. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  11. 2023Nash Learning from Human Feedbackcited by 220
  12. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  13. 2024Are We Done with MMLU?cited by 178
  14. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  15. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  16. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 121
  17. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 121
  18. 2024Can LLM be a Personalized Judge?cited by 99
  19. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  20. 2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 93
  21. 2024KMMLU: Measuring Massive Multitask Language Understanding in Koreancited by 91
  22. 2023ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agentcited by 78
  23. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  24. 2023HAE-RAE Bench: Evaluation of Korean Knowledge in Language Modelscited by 47
  25. 2022KOBEST: Korean Balanced Evaluation of Significant Taskscited by 37
  26. 2023Revealing the structure of language model capabilitiescited by 35
  27. 2024Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignmentcited by 26
  28. 2024EXAONE 3.0 7.8B Instruction Tuned Language Modelcited by 20
  29. 2024HyperCLOVA X Technical Reportcited by 18
  30. 2024Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Modelscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 15; 2026: 13.

32 papers.

Sort:

By area:

  1. 2024Are Your LLMs Capable of Stable Reasoning?cited by 65
  2. 2025Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gapcited by 41
  3. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 34
  4. 2025Heimdall: test-time scaling on the generative verificationcited by 26
  5. 2025When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Researchcited by 25
  6. 2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 21
  7. 2024GLIDER: Grading LLM Interactions and Decisions using Explainable Rankingcited by 14
  8. 2025Reinforcement Learning for Machine Learning Engineering Agentscited by 13
  9. 2024Generative Adversarial Reviews: When LLMs Become the Criticcited by 12
  10. 2025RecGPT Technical Reportcited by 11
  11. 2025RollArt: Disaggregated Multi-Task Agentic RL Training at Scalecited by 11
  12. 2026SWE-Replay: Efficient Test-Time Scaling for Software Engineering Agentscited by 11
20 more
  1. 2025ProRe: A Proactive Reward System for GUI Agents via Reasoner-Actor Collaborationcited by 10
  2. 2025GROVE: A Generalized Reward for Learning Open-Vocabulary Physical Skillcited by 9
  3. 2025AdaBoN: Adaptive Best-of-N Alignmentcited by 8
  4. 2026TimeWarp: Evaluating Web Agents by Revisiting the Pastcited by 6
  5. 2026Uncertainty-Aware Clarification in LLM Agents with Information Gaincited by 6
  6. 2025SkillVerse : Assessing and Enhancing LLMs with Tree Evaluationcited by 2
  7. 2025Shaping Explanations: Semantic Reward Modeling with Encoder-Only Transformers for GRPOcited by 2
  8. 2026Neglected Free Lunch from Post-training: Progress Advantage for LLM Agentscited by 2
  9. 2025OphthBench: A Comprehensive Benchmark for Evaluating Large Language Models in Chinese Ophthalmologycited by 1
  10. 2025Efficient Online RFT with Plug-and-Play LLM Judges: Unlocking State-of-the-Art Performancecited by 1
  11. 2026Judging What We Cannot Solve: A Consequence-Based Approach for Oracle-Free Evaluation of Research-Level Mathcited by 1
  12. 2026Bi-Level Prompt Optimization for Multimodal LLM-as-a-Judgecited by 1
  13. 2026Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systemscited by 1
  14. 2025Towards Reliable, Uncertainty-Aware Alignmentcited by 0
  15. 2026LLM-as-Judge for Semantic Judging of Powerline Segmentation in UAV Inspectioncited by 0
  16. 2026Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoringcited by 0
  17. 2026Data-driven and distributed governance of building facilities management using decentralized autonomous organization, digital twin, and large language modelscited by 0
  18. 2026From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Modelscited by 0
  19. 2026Towards Spec Learning: Inference-Time Alignment from Preference Pairscited by 0
  20. 2026Deep Interaction: An Efficient Human-AI Interaction Method for Large Reasoning Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Lower-Resource, Higher Scores: Language Bias in LLM Evaluators
  2. 2025Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments
  3. 2024Enhancing LLM Evaluations: The Garbling Trick
  4. 2026When Consistency Does Not Mean Reliability: Evaluating Local LLM Judges Against Human Ratings
  5. 2024Aligning Human and LLM Judgments: Insights from EvalAssist on Task-Specific Evaluations and AI-assisted Assessment Strategy Preferences
  6. 2023Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks
  7. 2025Evaluating Scoring Bias in LLM-as-a-Judge
  8. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge