mlsanityData to 14 Oct 2026

SAGEval: The frontiers of Satisfactory Agent based NLG Evaluation for reference-free open-ended text

arXiv 2411.16077 ↗First posted 25 Nov 2024cs.CL cs.MAcited by 1

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Large Language Model (LLM) integrations into applications like Microsoft365 suite and Google Workspace for creating/processing documents, emails, presentations, etc. has led to considerable enhancements in productivity and time savings. But as these integrations become more more complex, it is paramount to ensure that the quality of output from the LLM-integrated applications are relevant and appropriate for use. Identifying the need to develop robust evaluation approaches for natural language generation, wherein references/ground labels doesn't exist or isn't amply available, this paper introduces a novel framework called "SAGEval" which utilizes a critiquing Agent to provide feedback on scores generated by LLM evaluators. We show that the critiquing Agent is able to rectify scores from LLM evaluators, in absence of references/ground-truth labels, thereby reducing the need for labeled data even for complex NLG evaluation scenarios, like the generation of JSON-structured forms/surveys with responses in different styles like multiple choice, likert ratings, single choice questions, etc.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 24 of its 47 references that are in the map. First posted on arXiv 25 Nov 2024.

Builds on

24 of 47 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024MATEval: A Multi-Agent Discussion Framework for Advancing Open-Ended Text Evaluation28 Mar
  2. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors21 Aug
  3. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate14 Aug
  4. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework1 Aug

Other references · 20

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  6. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  7. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  8. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  9. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  10. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  11. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 378
  12. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
8 more
  1. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 106
  2. 2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 88
  3. 2023Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Studycited by 85
  4. 2023MetaAgents: Large Language Model Based Agents for Decision-Making on Teamingcited by 85
  5. 2021OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metricscited by 61
  6. 2024ToolNet: Connecting Large Language Models with Massive Tools via Tool Graphcited by 60
  7. 2023MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Taskscited by 57
  8. 2023X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspectscited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Agent-as-a-Judgecited by 17

Similar, not cited

Close in wording, no citation either way

  1. 2024DEBATE: Devil's Advocate-Based Assessment and Text Evaluation
  2. 2025Multi-Agent LLM Judge: automatic personalized LLM judge design for evaluating natural language generation applications
  3. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judges
  4. 2025Unbiased Evaluation of Large Language Models from a Causal Perspective
  5. 2025JudgeAgent: Beyond Static Benchmarks for Knowledge-Driven and Dynamic LLM Evaluation
  6. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
  7. 2026EvalBuilder: Can LLMs Build Efficient Benchmarks?
  8. 2023Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks