mlsanityData to 7 Oct 2026

Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences

arXiv 2404.12272 ↗First posted 18 Apr 2024cs.HC cs.AIcited by 142

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Due to the cumbersome nature of human evaluation and limitations of code-based evaluation, Large Language Models (LLMs) are increasingly being used to assist humans in evaluating LLM outputs. Yet LLM-generated evaluators simply inherit all the problems of the LLMs they evaluate, requiring further human validation. We present a mixed-initiative approach to ``validate the validators'' -- aligning LLM-generated evaluation functions (be it prompts or code) with human requirements. Our interface, EvalGen, provides automated assistance to users in generating evaluation criteria and implementing assertions. While generating candidate implementations (Python functions, LLM grader prompts), EvalGen asks humans to grade a subset of LLM outputs; this feedback is used to select implementations that better align with user grades. A qualitative study finds overall support for EvalGen but underscores the subjectivity and iterative process of alignment. In particular, we identify a phenomenon we dub criteria drift: users need criteria to grade outputs, but grading outputs helps users define criteria. What is more, some criteria appears dependent on the specific LLM outputs observed (rather than independent criteria that can be defined a priori), raising serious questions for approaches that assume the independence of evaluation from observation of model outputs. We present our interface and implementation details, a comparison of our algorithm with a baseline approach, and implications for the design of future LLM evaluation assistants.

Key facts. Cited by 142 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 64 references that are in the map. First posted on arXiv 18 Apr 2024.

Builds on

27 of 64 references.

References · 27

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  4. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  5. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  6. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  7. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  8. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
  9. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 306
  10. 2023How is ChatGPT's behavior changing over time?cited by 265
  11. 2024Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoderscited by 213
  12. 2022PromptChainer: Chaining Large Language Model Prompts through Visual Programmingcited by 134
15 more
  1. 2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 119
  2. 2021To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-makingcited by 118
  3. 2023ACI-BENCH: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generationcited by 116
  4. 2022Explanations Can Reduce Overreliance on AI Systems During Decision-Makingcited by 111
  5. 2023ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testingcited by 88
  6. 2023EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteriacited by 73
  7. 2024AutoEval Done Right: Using Synthetic Data for Model Evaluationcited by 53
  8. 2022Designing for Responsible Trust in AI Systems: A Communication Perspectivecited by 48
  9. 2023Supporting Human-AI Collaboration in Auditing LLMs with LLMscited by 46
  10. 2023ConstitutionMaker: Interactively Critiquing Large Language Models by Converting Feedback into Principlescited by 33
  11. 2023DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelinescited by 27
  12. 2024Supporting Sensemaking of Large Language Model Outputs at Scalecited by 26
  13. 2023"AI enhances our performance, I have no doubt this one will do the same": The Placebo effect is robust to negative descriptions of AIcited by 11
  14. 2014ICE: Enabling Non-Experts to Build Models Interactively for Large-Scale Lopsided Problemscited by 7
  15. 2023Co-audit: tools to help humans double-check AI-generated contentcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 60; 2026: 61.

142 papers.

Sort:

By area:

  1. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
  2. 2024Can LLM be a Personalized Judge?cited by 99
  3. 2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Futurecited by 85
  4. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  5. 2025Interactive Debugging and Steering of Multi-Agent AI Systemscited by 68
  6. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  7. 2024DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processingcited by 49
  8. 2025Creating General User Models from Computer Usecited by 44
  9. 2024Inverse Constitutional AI: Compressing Preferences into Principlescited by 39
  10. 2025Optimizing Model Selection for Compound AI Systemscited by 34
  11. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 32
  12. 2025What Prompts Don't Say: Understanding and Managing Underspecification in LLM Promptscited by 32
38 more
  1. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  2. 2024What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Usecited by 24
  3. 2026Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judgescited by 23
  4. 2025Multi-Agent Debate for LLM Judges with Adaptive Stability Detectioncited by 20
  5. 2025VeriPlan: Integrating Formal Verification and LLMs into End-User Planningcited by 19
  6. 2026Learning to Judge: LLMs Designing and Applying Evaluation Rubricscited by 18
  7. 2025GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluationcited by 17
  8. 2024Regurgitative Training: The Value of Real Data in Training Large Language Modelscited by 15
  9. 2024SciDoc2Diagrammer-MAF: Towards Generation of Scientific Diagrams from Documents guided by Multi-Aspect Feedback Refinementcited by 15
  10. 2024Beyond the Comfort Zone: Emerging Solutions to Overcome Challenges in Integrating LLMs into Software Productscited by 11
  11. 2025A Scalable Framework for Evaluating Health Language Modelscited by 11
  12. 2026An Imperfect Verifier is Good Enough: Learning with Noisy Rewardscited by 11
  13. 2025SPHERE: An Evaluation Card for Human-AI Systemscited by 9
  14. 2025LLMs for Customized Marketing Content Generation and Evaluation at Scalecited by 9
  15. 2025Verification-Aware Planning for Multi-Agent Systemscited by 9
  16. 2025Reducing Large Language Model Safety Risks in Women's Health using Semantic Entropycited by 8
  17. 2025From Hazard Identification to Controller Design: Proactive and LLM-Supported Safety Engineering for ML-Powered Systemscited by 7
  18. 2025Addressing Bias in Generative AI: Challenges and Research Opportunities in Information Managementcited by 7
  19. 2025Verdict: A Library for Scaling Judge-Time Computecited by 7
  20. 2025Unequal Opportunities: Examining the Bias in Geographical Recommendations by Large Language Modelscited by 7
  21. 2025Benchmarking Multi-National Value Alignment for Large Language Modelscited by 7
  22. 2025Just-In-Time Objectives: A General Approach for Specialized AI Interactionscited by 7
  23. 2026DiLLS: Interactive Diagnosis of LLM-based Multi-agent Systems via Layered Summary of Agent Behaviorscited by 7
  24. 2026Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelinescited by 7
  25. 2026The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluationcited by 7
  26. 2024Sociotechnical Implications of Generative Artificial Intelligence for Information Accesscited by 6
  27. 2024ChainBuddy: An AI Agent System for Generating LLM Pipelinescited by 6
  28. 2025PairBench: Are Vision-Language Models Reliable at Comparing What They See?cited by 6
  29. 2025A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Userscited by 6
  30. 2026AgentFixer: From Failure Detection to Fix Recommendations in LLM Agentic Systemscited by 6
  31. 2024Policy Maps: Tools for Guiding the Unbounded Space of LLM Behaviorscited by 5
  32. 2024Visual Prompting with Iterative Refinement for Design Critique Generationcited by 5
  33. 2025PROMPTEVALS: A Dataset of Assertions and Guardrails for Custom Production Large Language Model Pipelinescited by 5
  34. 2025Uncovering Systematic Failures of LLMs in Verifying Code Against Natural Language Specificationscited by 5
  35. 2026Are LLMs Reliable Code Reviewers? Systematic Overcorrection in Requirement Conformance Judgementcited by 4
  36. 2024The Interaction Layer: An Exploration for Co-Designing User-LLM Interactions in Parental Wellbeing Support Systemscited by 3
  37. 2024A Framework for Evaluating LLMs Under Task Indeterminacycited by 3
  38. 2025Semantic Commit: Helping Users Update Intent Specifications for AI Memory at Scalecited by 3

50 most cited of 142 All 142 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Aligning Human and LLM Judgments: Insights from EvalAssist on Task-Specific Evaluations and AI-assisted Assessment Strategy Preferences
  2. 2025An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliability
  3. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
  4. 2025On Evaluating LLM Alignment by Evaluating LLMs as Judges
  5. 2023Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks
  6. 2024Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructions
  7. 2025RocketEval: Efficient Automated LLM Evaluation via Grading Checklist
  8. 2024Evaluating the Consistency of LLM Evaluators