Assessing the Performance of Human-Capable LLMs -- Are LLMs Coming for Your Job?
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks
Abstract (arXiv)
The current paper presents the development and validation of SelfScore, a novel benchmark designed to assess the performance of automated Large Language Model (LLM) agents on help desk and professional consultation tasks. Given the increasing integration of AI in industries, particularly within customer service, SelfScore fills a crucial gap by enabling the comparison of automated agents and human workers. The benchmark evaluates agents on problem complexity and response helpfulness, ensuring transparency and simplicity in its scoring system. The study also develops automated LLM agents to assess SelfScore and explores the benefits of Retrieval-Augmented Generation (RAG) for domain-specific tasks, demonstrating that automated LLM agents incorporating RAG outperform those without. All automated LLM agents were observed to perform better than the human control group. Given these results, the study raises concerns about the potential displacement of human workers, especially in areas where AI technologies excel. Ultimately, SelfScore provides a foundational tool for understanding the impact of AI in help desk environments while advocating for ethical considerations in the ongoing transition towards automation.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 10 of its 44 references that are in the map. First posted on arXiv 5 Oct 2024.
Builds on
10 of 44 references.
Closest ancestors · 1
Same direction, newest first
Other references · 9
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
- 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
- 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
- 2023Large Language Models for Generative Information Extraction: A Surveycited by 183
- 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 123
- 2023Evaluating Cognitive Maps and Planning in Large Language Models with CogEvalcited by 77
- 2024Easy Problems That LLMs Get Wrongcited by 29
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMs
- 2025LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey
- 2024A Survey of Useful LLM Evaluation
- 2024Automated test generation to evaluate tool-augmented LLMs as conversational AI agents
- 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks
- 2023Who's Thinking? A Push for Human-Centered Evaluation of LLMs using the XAI Playbook
- 2023Exploring Qualitative Research Using LLMs
- 2023Enhancing Trust in LLM-Based AI Automation Agents: New Considerations and Future Challenges