mlsanityData to 7 Oct 2026

Learning to Align Multi-Faceted Evaluation: A Unified and Robust Framework

arXiv 2502.18874 ↗First posted 26 Feb 2025cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are being used more and more extensively for automated evaluation in various scenarios. Previous studies have attempted to fine-tune open-source LLMs to replicate the evaluation explanations and judgments of powerful proprietary models, such as GPT-4. However, these methods are largely limited to text-based analyses under predefined general criteria, resulting in reduced adaptability for unseen instructions and demonstrating instability in evaluating adherence to quantitative and structural constraints. To address these limitations, we propose a novel evaluation framework, ARJudge, that adaptively formulates evaluation criteria and synthesizes both text-based and code-driven analyses to evaluate LLM responses. ARJudge consists of two components: a fine-tuned Analyzer that generates multi-faceted evaluation analyses and a tuning-free Refiner that combines and refines all analyses to make the final judgment. We construct a Composite Analysis Corpus that integrates tasks for evaluation criteria generation alongside text-based and code-driven analysis generation to train the Analyzer. Our results demonstrate that ARJudge outperforms existing fine-tuned evaluators in effectiveness and robustness. Furthermore, it demonstrates the importance of multi-faceted evaluation and code-driven analyses in enhancing evaluation capabilities.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 42 references that are in the map. First posted on arXiv 26 Feb 2025.

Builds on

29 of 42 references.

References · 29

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2024Qwen2.5 Technical Reportcited by 4,713
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  7. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  8. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  9. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  10. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  11. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  12. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
17 more
  1. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  2. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  3. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  4. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  5. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  6. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
  7. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  8. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 104
  9. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 90
  10. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  11. 2023CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generationcited by 65
  12. 2023ICE-Score: Instructing Large Language Models to Evaluate Codecited by 50
  13. 2023Towards Reasoning in Large Language Models via Multi-Agent Peer Review Collaborationcited by 48
  14. 2023X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspectscited by 28
  15. 2024Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretabilitycited by 19
  16. 2024RevisEval: Improving LLM-as-a-Judge via Response-Adapted Referencescited by 19
  17. 2023Little Giants: Exploring the Potential of Small LLMs as Evaluation Metrics in Summarization in the Eval4NLP 2023 Shared Taskcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
  2. 2024Aligning Human and LLM Judgments: Insights from EvalAssist on Task-Specific Evaluations and AI-assisted Assessment Strategy Preferences
  3. 2024CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolution
  4. 2024Optimizing Alignment with Less: Leveraging Data Augmentation for Personalized Evaluation
  5. 2023A Closer Look into Automatic Evaluation Using Large Language Models
  6. 2024Enhancing LLM Evaluations: The Garbling Trick
  7. 2025An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliability
  8. 2025Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments