mlsanityData to 14 Oct 2026

Large Language Models Are Active Critics in NLG Evaluation

arXiv 2410.10724 ↗First posted 14 Oct 2024cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

The conventional paradigm of using large language models (LLMs) for natural language generation (NLG) evaluation relies on pre-defined task definitions and evaluation criteria, positioning LLMs as "passive critics" that strictly follow developer-provided guidelines. However, human evaluators often apply implicit criteria, and their expectations in practice can vary widely based on specific end-user needs. Consequently, these rigid evaluation methods struggle to adapt to diverse scenarios without extensive prompt customization. To address this, we introduce Active-Critic, a novel LLM-based evaluator that transforms LLMs into "active critics'' capable of adapting to diverse NLG tasks using limited example data. Active-Critic consists of two stages: (1) self-inferring the target NLG task and relevant evaluation criteria, and (2) dynamically optimizing prompts to produce human-aligned scores along with detailed justifications. Our experiments show that Active-Critic can generate nuanced, context-aware evaluation criteria, enabling it to achieve superior alignment with human judgments across multiple tasks.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 43 of its 61 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

43 of 61 references.

References · 43

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  5. 2015Teaching Machines to Read and Comprehendcited by 1,049
  6. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  7. 2023Large Language Models as Optimizerscited by 864
  8. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  9. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  10. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  11. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  12. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
31 more
  1. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  2. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  3. 2023GPTScore: Evaluate as You Desirecited by 361
  4. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  5. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 294
  6. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 291
  7. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  8. 2020Evaluation of Text Generation: A Surveycited by 277
  9. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  10. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 257
  11. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 188
  12. 2023Generative Judge for Evaluating Alignmentcited by 169
  13. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 151
  14. 2015Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systemscited by 134
  15. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 101
  16. 2023Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Methodcited by 76
  17. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 73
  18. 2023TIGERScore: Towards Building Explainable Metric for All Text Generation Taskscited by 70
  19. 2021OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metricscited by 61
  20. 2019TIGEr: Text-to-Image Grounding for Image Caption Evaluationcited by 58
  21. 2023Calibrating LLM-Based Evaluatorcited by 53
  22. 2018Sentence-Level Fluency Evaluation: References Help, But Can Be Spared!cited by 43
  23. 2023Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theorycited by 43
  24. 2024HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decompositioncited by 33
  25. 2023Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluationcited by 24
  26. 2023Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Modelcited by 21
  27. 2024Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretabilitycited by 19
  28. 2023BatchEval: Towards Human-like Text Evaluationcited by 17
  29. 2023CoAScore: Chain-of-Aspects Prompting for NLG Evaluationcited by 14
  30. 2024Intent-based Prompt Calibration: Enhancing prompt optimization with synthetic boundary casescited by 13
  31. 2023Exploring Prompting Large Language Models as Explainable Metricscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 1.

6 papers.

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  3. 2025Exploring the Multilingual NLG Evaluation Abilities of LLM-Based Evaluatorscited by 5
  4. 2025Automated Refinement of Essay Scoring Rubrics for Language Models via Reflect-and-Revisecited by 4
  5. 2025Beyond Task-Oriented and Chitchat Dialogues: Proactive and Transition-Aware Conversational Agentscited by 2
  6. 2026CE-RM: A Pointwise Generative Reward Model Optimized via Two-Stage Rollout and Unified Criteriacited by 2