Beyond Benchmarks: LLM Evaluation with an Anthropomorphic and Lifecycle-oriented Roadmap
Not on the map. Why?
Abstract (arXiv)
Despite their rapid advancement, large language models (LLMs) suffer from a critical disconnect between benchmark scores and real-world utility. Current evaluation remains fragmented, prioritizing isolated technical metrics over the holistic, developmental, and societal aspects essential for deployment. Rather than serving merely as a descriptive catalog, this work establishes a diagnostic ontology that causally maps evaluation dimensions to the canonical LLM training pipeline, transforming evaluation from static ranking into a diagnostic tool for root-cause analysis. In this paper, we introduce an anthropomorphic evaluation framework that re-conceptualizes LLM capabilities through a four-dimensional lens: Intelligence Quotient (IQ), Professional Quotient (PQ), Emotional Quotient (EQ), and Value-oriented Quotient (VQ). We operationalize these concepts through a modular evaluation architecture and validate the framework's diagnostic claims through meta-analysis of public benchmark trends. Analyzing over 200 benchmarks, we synthesize key challenges and future directions. This work offers a strategic compass for developing LLMs that are not only technically proficient but also contextually relevant and ethically sound. A curated repository is available at: https://github.com/onejune2018/Awesome-LLM-Eval.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 167 of its 210 references that are in the map. First posted on arXiv 26 Aug 2025.
Builds on
167 of 210 references.
References · 167
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
38 more
- 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
- 2021Program Synthesis with Large Language Modelscited by 3,637
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
- 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
- 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
- 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
- 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
- 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
- 2015A large annotated corpus for learning natural language inferencecited by 1,847
- 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
- 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
- 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
- 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
- 2024A Survey on LLM-as-a-Judgecited by 1,423
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
- 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
- 2023A Survey on Evaluation of Large Language Modelscited by 1,242
- 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
- 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
- 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
- 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
- 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
- 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
- 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
- 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
- 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
- 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
50 most cited of 167 All 167 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026Evaluating Cognitive Age Alignment in Interactive AI Agents18 May 2026