mlsanityData to 7 Oct 2026

Assessing and Verifying Task Utility in LLM-Powered Applications

arXiv 2405.02178 ↗First posted 3 May 2024cs.CL cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

The rapid development of Large Language Models (LLMs) has led to a surge in applications that facilitate collaboration among multiple agents, assisting humans in their daily tasks. However, a significant gap remains in assessing to what extent LLM-powered applications genuinely enhance user experience and task execution efficiency. This highlights the need to verify utility of LLM-powered applications, particularly by ensuring alignment between the application's functionality and end-user needs. We introduce AgentEval, a novel framework designed to simplify the utility verification process by automatically proposing a set of criteria tailored to the unique purpose of any given application. This allows for a comprehensive assessment, quantifying the utility of an application against the suggested criteria. We present a comprehensive analysis of the effectiveness and robustness of AgentEval for two open source datasets including Math Problem solving and ALFWorld House-hold related tasks. For reproducibility purposes, we make the data, code and all the logs publicly available at https://bit.ly/3w3yKcS .

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 36 of its 61 references that are in the map. First posted on arXiv 3 May 2024.

Builds on

36 of 61 references.

References · 36

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  5. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  6. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  7. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  8. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  9. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  10. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  11. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  12. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
24 more
  1. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  2. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  3. 2020Aligning AI With Shared Human Valuescited by 706
  4. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  5. 2023Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agentscited by 329
  6. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  7. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
  8. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  9. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 176
  10. 2019What makes a good conversation? How controllable attributes affect human judgmentscited by 140
  11. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 136
  12. 2024Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Codecited by 106
  13. 2023Multi-Dimensional Evaluation of Text Summarization with In-Context Learningcited by 44
  14. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debatecited by 40
  15. 2023Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Controlcited by 32
  16. 2023Creative Agents: Empowering Agents with Imagination for Creative Taskscited by 24
  17. 2022Interactive Grounded Language Understanding in a Collaborative Environment: IGLU 2021cited by 23
  18. 2023Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Taskscited by 23
  19. 2022Aligning Offline Metrics and Human Judgments of Value for Code Generation Modelscited by 14
  20. 2022Semantic-Oriented Unlabeled Priming for Large-Scale Language Modelscited by 12
  21. 2023From Voices to Validity: Leveraging Large Language Models (LLMs) for Textual Analysis of Policy Stakeholder Interviewscited by 8
  22. 2021DEUS: A Data-driven Approach to Estimate User Satisfaction in Multi-turn Dialoguescited by 7
  23. 2017Evaluating Personal Assistants on Mobile devicescited by 3
  24. 2020Optimizing Interactive Systems via Data-Driven Objectivescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 7; 2026: 4.

15 papers.

Sort:

By area:

  1. 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
  2. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 98
  3. 2025Benchmarking Prompt Sensitivity in Large Language Modelscited by 38
  4. 2025DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systemscited by 23
  5. 2024Towards Effective GenAI Multi-Agent Collaboration: Design and Evaluation for Enterprise Applicationscited by 19
  6. 2025A Human-AI Comparative Analysis of Prompt Sensitivity in LLM-Based Relevance Judgmentcited by 15
  7. 2025Benchmarking LLM-based Relevance Judgment Methodscited by 14
  8. 2024BenchAgents: Multi-Agent Systems for Structured Benchmark Creationcited by 6
  9. 2026Tracing the Data Trail: A Survey of Data Provenance, Transparency and Traceability in LLMscited by 3
  10. 2026The Hitchhikers Guide to Rubric Quality Understanding and Enrichmentcited by 3
  11. 2024LLM-Human Pipeline for Cultural Context Grounding of Conversationscited by 2
  12. 2026PReMISE: Policy Rubrics as Measurement Specifications for LLM Judgescited by 2
3 more
  1. 2024IDAT: A Multi-Modal Dataset and Toolkit for Building and Evaluating Interactive Task-Solving Agentscited by 1
  2. 2025PersonaMatrix: A Recipe for Persona-Aware Evaluation of Legal Summarizationcited by 0
  3. 2026From Performance to Purpose: A Sociotechnical Taxonomy for Evaluating Large Language Model Utilitycited by 0