mlsanityData to 7 Oct 2026

A Survey of Useful LLM Evaluation

arXiv 2406.00936 ↗First posted 3 Jun 2024cs.CLcited by 28

Not on the map. Why?

Abstract (arXiv)

LLMs have gotten attention across various research domains due to their exceptional performance on a wide range of complex tasks. Therefore, refined methods to evaluate the capabilities of LLMs are needed to determine the tasks and responsibility they should undertake. Our study mainly discussed how LLMs, as useful tools, should be effectively assessed. We proposed the two-stage framework: from ``core ability'' to ``agent'', clearly explaining how LLMs can be applied based on their specific capabilities, along with the evaluation methods in each stage. Core ability refers to the capabilities that LLMs need in order to generate high-quality natural language texts. After confirming LLMs possess core ability, they can solve real-world and complex tasks as agent. In the "core ability" stage, we discussed the reasoning ability, societal impact, and domain knowledge of LLMs. In the ``agent'' stage, we demonstrated embodied action, planning, and tool learning of LLMs agent applications. Finally, we examined the challenges currently confronting the evaluation methods for LLMs, as well as the directions for future development.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 103 of its 168 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

103 of 168 references.

References · 103

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  10. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
38 more
  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  2. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  3. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  5. 2019Language Models as Knowledge Bases?cited by 1,922
  6. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  7. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  8. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  10. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  11. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  12. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  13. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  14. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  15. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  16. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  17. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  18. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  19. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  20. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  21. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  22. 2019ELI5: Long Form Question Answeringcited by 567
  23. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 476
  24. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  25. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  26. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  27. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  28. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 378
  29. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  30. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  31. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  32. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  33. 2019Abductive Commonsense Reasoningcited by 300
  34. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  35. 2023Large Language Models as Tool Makerscited by 293
  36. 2022Teaching language models to support answers with verified quotescited by 290
  37. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  38. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278

50 most cited of 103 All 103 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 18; 2026: 4.

28 papers.

Sort:

By area:

  1. 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
  2. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 424
  3. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  4. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
  5. 2024What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluationcited by 17
  6. 2025Long Context Scaling: Divide and Conquer via Multi-Agent Question-driven Collaborationcited by 7
  7. 2024Open Ko-LLM Leaderboard2: Bridging Foundational and Practical Evaluation for Korean LLMscited by 6
  8. 2025MedOmni-45deg: A Safety-Performance Benchmark for Reasoning-Oriented LLMs in Medicinecited by 6
  9. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Modelscited by 5
  10. 2024TestAgent: Automatic Benchmarking and Exploratory Interaction for Evaluating LLMs in Vertical Domainscited by 4
  11. 2025Noise, Adaptation, and Strategy: Assessing LLM Fidelity in Decision-Makingcited by 4
  12. 2025A Principle-based Framework for the Development and Evaluation of Large Language Models for Health and Wellnesscited by 3
16 more
  1. 2026Rethinking Role-Playing Evaluation: Anonymous Benchmarking and a Systematic Study of Personality Effectscited by 3
  2. 2024A Survey on Complex Tasks for Goal-Directed Interactive Agentscited by 2
  3. 2024Representing the Under-Represented: Cultural and Core Capability Benchmarks for Developing Thai Large Language Modelscited by 2
  4. 2025Cross-Entropy Games for Language Models: From Implicit Knowledge to General Capability Measurescited by 2
  5. 2025Can LLM Assist in the Evaluation of the Quality of Machine Learning Explanations?cited by 1
  6. 2025Model Consistency as a Cheap yet Predictive Proxy for LLM Elo Scorescited by 1
  7. 2025Benchmarking Open-Source Large Language Models for Persian in Zero-Shot and Few-Shot Learningcited by 1
  8. 2025If I Could Turn Back Time: Temporal Reframing as a Historical Reasoning Task for LLMscited by 1
  9. 2026Key Considerations for Domain Expert Involvement in LLM Design and Evaluation: An Ethnographic Studycited by 1
  10. 2024QUENCH: Measuring the gap between Indic and Non-Indic Contextual General Reasoning in LLMscited by 0
  11. 2025Make a Video Call with LLM: A Measurement Campaign over Six Mainstream Appscited by 0
  12. 2025Validity Is What You Needcited by 0
  13. 2025Conversational No-code, Multi-agentic Disease Module Identification and Drug Repurposing Prediction with ChatDRexcited by 0
  14. 2025Towards Privacy-Preserving Code Generation: Differentially Private Code Language Modelscited by 0
  15. 2026Beyond Aggregate Scores: Behavioral Correctness Assumptions for Assessing Reference-Based Automatic Evaluation Methodscited by 0
  16. 2026When Passing Tests Hides Vulnerabilities: An Empirical Study of Silent Failures in Agentic Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023AgentBench: Evaluating LLMs as Agents
  2. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey
  3. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
  4. 2025When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMs
  5. 2025Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments
  6. 2024Beyond Natural Language: LLMs Leveraging Alternative Formats for Enhanced Reasoning and Communication
  7. 2023Enhancing Open-Domain Task-Solving Capability of LLMs via Autonomous Tool Integration from GitHub
  8. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judges