mlsanityData to 14 Oct 2026

Prometheus: Inducing Fine-grained Evaluation Capability in Language Models

arXiv 2310.08491 ↗First posted 12 Oct 2023cs.CL cs.LGcited by 575

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Recently, using a powerful proprietary Large Language Model (LLM) (e.g., GPT-4) as an evaluator for long-form responses has become the de facto standard. However, for practitioners with large-scale evaluation tasks and custom criteria in consideration (e.g., child-readability), using proprietary LLMs as an evaluator is unreliable due to the closed-source nature, uncontrolled versioning, and prohibitive costs. In this work, we propose Prometheus, a fully open-source LLM that is on par with GPT-4's evaluation capabilities when the appropriate reference materials (reference answer, score rubric) are accompanied. We first construct the Feedback Collection, a new dataset that consists of 1K fine-grained score rubrics, 20K instructions, and 100K responses and language feedback generated by GPT-4. Using the Feedback Collection, we train Prometheus, a 13B evaluator LLM that can assess any given long-form text based on customized score rubric provided by the user. Experimental results show that Prometheus scores a Pearson correlation of 0.897 with human evaluators when evaluating with 45 customized score rubrics, which is on par with GPT-4 (0.882), and greatly outperforms ChatGPT (0.392). Furthermore, measuring correlation with GPT-4 with 1222 customized score rubrics across four benchmarks (MT Bench, Vicuna Bench, Feedback Bench, Flask Eval) shows similar trends, bolstering Prometheus's capability as an evaluator LLM. Lastly, Prometheus achieves the highest accuracy on two human preference benchmarks (HHH Alignment & MT Bench Human Judgment) compared to open-sourced reward models explicitly trained on human preference datasets, highlighting its potential as an universal reward model. We open-source our code, dataset, and model at https://kaistai.github.io/prometheus/.

Key facts. Cited by 575 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 31 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

26 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback22 May

Other references · 25

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  6. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  7. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  8. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  9. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  10. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  11. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  12. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
13 more
  1. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  2. 2023Instruction Tuning with GPT-4cited by 686
  3. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  4. 2022Large Language Models Are Reasoning Teacherscited by 329
  5. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  6. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 201
  7. 2021Hurdles to Progress in Long-form Question Answeringcited by 167
  8. 2023The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuningcited by 108
  9. 2023Shepherd: A Critic for Language Model Generationcited by 97
  10. 2023Aligning Large Language Models through Synthetic Feedbackcited by 75
  11. 2023INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Modelscited by 62
  12. 2023On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Researchcited by 44
  13. 2023Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?cited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 129; 2025: 168; 2026: 269.

575 papers.

Sort:

By area:

  1. 2024Self-Rewarding Language Modelscited by 681
  2. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  3. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  4. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  5. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  6. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  7. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  8. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  9. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  10. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
  11. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  12. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
38 more
  1. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 158
  2. 2024Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Modelscited by 154
  3. 2024Generative Reward Modelscited by 148
  4. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 129
  5. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 125
  6. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 125
  7. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 123
  8. 2025Process Reward Models That Thinkcited by 118
  9. 2024Critique-out-Loud Reward Modelscited by 106
  10. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 104
  11. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revisioncited by 98
  12. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 94
  13. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 94
  14. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  15. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  16. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedbackcited by 90
  17. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  18. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 78
  19. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 75
  20. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 74
  21. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 74
  22. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 73
  23. 2024Self-Taught Evaluatorscited by 72
  24. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 71
  25. 2023AlignBench: Benchmarking Chinese Alignment of Large Language Modelscited by 68
  26. 2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 64
  27. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 64
  28. 2025Evaluating Step-by-step Reasoning Traces: A Surveycited by 63
  29. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 62
  30. 2024Quantile Regression for Distributional Reward Models in RLHFcited by 60
  31. 2025One Token to Fool LLM-as-a-Judgecited by 60
  32. 2024CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferencescited by 57
  33. 2024Lynx: An Open Source Hallucination Evaluation Modelcited by 57
  34. 2024FinTextQA: A Dataset for Long-form Financial Question Answeringcited by 54
  35. 2024Can Unconfident LLM Annotations Be Used for Confident Conclusions?cited by 49
  36. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 47
  37. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 45
  38. 2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 43

50 most cited of 575 All 575 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
  2. 2026EvalBuilder: Can LLMs Build Efficient Benchmarks?
  3. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judges
  4. 2025Accelerating Unbiased LLM Evaluation via Synthetic Feedback
  5. 2025LLMs Are Not Scorers: Rethinking MT Evaluation with Generation-Based Methods
  6. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback
  7. 2026LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation
  8. 2026Mitigating Scoring Bias in LLM-as-a-Judge via Random Number Generation