mlsanityData to 13 Oct 2026

Evaluating Large Language Models at Evaluating Instruction Following

arXiv 2310.07641 ↗First posted 11 Oct 2023cs.CL cs.LGcited by 297

Not on the map. Why?

Cited by 61 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

As research in large language models (LLMs) continues to accelerate, LLM-based evaluation has emerged as a scalable and cost-effective alternative to human evaluations for comparing the ever increasing list of models. This paper investigates the efficacy of these ``LLM evaluators'', particularly in using them to assess instruction following, a metric that gauges how closely generated text adheres to the given instruction. We introduce a challenging meta-evaluation benchmark, LLMBar, designed to test the ability of an LLM evaluator in discerning instruction-following outputs. The authors manually curated 419 pairs of outputs, one adhering to instructions while the other diverging, yet may possess deceptive qualities that mislead an LLM evaluator, e.g., a more engaging tone. Contrary to existing meta-evaluation, we discover that different evaluators (i.e., combinations of LLMs and prompts) exhibit distinct performance on LLMBar and even the highest-scoring ones have substantial room for improvement. We also present a novel suite of prompting strategies that further close the gap between LLM and human evaluators. With LLMBar, we hope to offer more insight into LLM evaluators and foster future research in developing better instruction-following models.

Key facts. Cited by 297 papers in the map (data through 13 Oct 2026). Builds on 52 of its 67 references that are in the map. First posted on arXiv 11 Oct 2023.

Builds on

52 of 67 references.

References · 52

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2023GPT-4 Technical Reportcited by 18,278
  3. 2022Training language models to follow instructions with human feedbackcited by 15,856
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,376
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  10. 2017Deep reinforcement learning from human preferencescited by 3,871
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,693
  12. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,228
38 more
  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,985
  2. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,758
  3. 2023PaLM 2 Technical Reportcited by 1,162
  4. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,120
  5. 2023LIMA: Less Is More for Alignmentcited by 1,091
  6. 2023Large Language Models are not Fair Evaluatorscited by 1,054
  7. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,041
  8. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 745
  9. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 734
  10. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 731
  11. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 717
  12. 2023Instruction Tuning with GPT-4cited by 681
  13. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 617
  14. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 421
  15. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  16. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  17. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 400
  18. 2023GPTScore: Evaluate as You Desirecited by 361
  19. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 330
  20. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 313
  21. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  22. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 287
  23. 2023Preference Ranking Optimization for Human Alignmentcited by 284
  24. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 260
  25. 2023The False Promise of Imitating Proprietary LLMscited by 258
  26. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 244
  27. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 241
  28. 2023Self-Alignment with Instruction Backtranslationcited by 149
  29. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  30. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  31. 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 96
  32. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 92
  33. 2021Understanding by Understanding Not: Modeling Negation in Language Modelscited by 73
  34. 2023COLLIE: Systematic Construction of Constrained Text Generation Taskscited by 69
  35. 2022Can Large Language Models Truly Understand Prompts? A Case Study with Negated Promptscited by 57
  36. 2022An Analysis of Negation in Natural Language Understanding Corporacited by 39
  37. 2023Instruction-following Evaluation through Verbalizer Manipulationcited by 30
  38. 2023The Trickle-down Impact of Reward (In-)consistency on RLHFcited by 29

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 5; 2024: 116; 2025: 96; 2026: 80.

297 papers.

Sort:

By area:

  1. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 488
  2. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 483
  3. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 324
  4. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 317
  5. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 304
  6. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 258
  7. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 239
  8. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 201
  9. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 195
  10. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  11. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 175
  12. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 172
38 more
  1. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 153
  2. 2024Generative Reward Modelscited by 148
  3. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 147
  4. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 142
  5. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 141
  6. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
  7. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 130
  8. 2023FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Modelscited by 116
  9. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Modelscited by 109
  10. 2024DiscoveryBench: Towards Data-Driven Discovery with Large Language Modelscited by 106
  11. 2023Graph Foundation Models: Concepts, Opportunities and Challengescited by 101
  12. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 101
  13. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 100
  14. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Modelscited by 98
  15. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  16. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 90
  17. 2024Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuningcited by 85
  18. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  19. 2024How to Evaluate Reward Models for RLHFcited by 84
  20. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 77
  21. 2024Self-Taught Evaluatorscited by 70
  22. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 70
  23. 2023Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarizationcited by 69
  24. 2024FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructionscited by 68
  25. 2024SafeBench: A Safety Evaluation Framework for Multimodal Large Language Modelscited by 68
  26. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  27. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 60
  28. 2024Measuring and Controlling Instruction (In)Stability in Language Model Dialogscited by 53
  29. 2024LatentQA: Teaching LLMs to Decode Activations Into Natural Languagecited by 51
  30. 2024Evaluating LLMs at Detecting Errors in LLM Responsescited by 50
  31. 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 50
  32. 2024Preference Learning Algorithms Do Not Learn Preference Rankingscited by 48
  33. 2024Improving Reward Models with Synthetic Critiquescited by 44
  34. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluatorscited by 44
  35. 2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 43
  36. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
  37. 2025Tower+: Bridging Generality and Translation Specialization in Multilingual LLMscited by 40
  38. 2025Do LLM Evaluators Prefer Themselves for a Reason?cited by 38

50 most cited of 297 All 297 (JSON)