mlsanityData to 7 Oct 2026

Evaluating AI Evaluation: Perils and Prospects

arXiv 2407.09221 ↗First posted 12 Jul 2024cs.AI cs.CYcited by 11

Not on the map. Why?

Abstract (arXiv)

As AI systems appear to exhibit ever-increasing capability and generality, assessing their true potential and safety becomes paramount. This paper contends that the prevalent evaluation methods for these systems are fundamentally inadequate, heightening the risks and potential hazards associated with AI. I argue that a reformation is required in the way we evaluate AI systems and that we should look towards cognitive sciences for inspiration in our approaches, which have a longstanding tradition of assessing general intelligence across diverse species. We will identify some of the difficulties that need to be overcome when applying cognitively-inspired approaches to general-purpose AI systems and also analyse the emerging area of "Evals". The paper concludes by identifying promising research pathways that could refine AI evaluation, advancing it towards a rigorous scientific domain that contributes to the development of safe AI systems.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 54 of its 154 references that are in the map. First posted on arXiv 12 Jul 2024.

Builds on

54 of 154 references.

References · 54

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  9. 2016Concrete Problems in AI Safetycited by 1,582
  10. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  11. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  12. 2015Teaching Machines to Read and Comprehendcited by 1,039
38 more
  1. 2021Ethical and social risks of harm from Language Modelscited by 957
  2. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  3. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  4. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  5. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  6. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  7. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  8. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  9. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  10. 2020Artificial Intelligence, Values and Alignmentcited by 443
  11. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  12. 2022Using cognitive psychology to understand GPT-3cited by 389
  13. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  14. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  15. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  16. 2023An Overview of Catastrophic AI Riskscited by 214
  17. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  18. 2017Deal or No Deal? End-to-End Learning for Negotiation Dialoguescited by 202
  19. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
  20. 2021Alignment of Language Agentscited by 167
  21. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  22. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 134
  23. 2019Behaviour Suite for Reinforcement Learningcited by 129
  24. 2024Evaluating Frontier Models for Dangerous Capabilitiescited by 121
  25. 2019Optimal Policies Tend to Seek Powercited by 115
  26. 2017AI Safety Gridworldscited by 108
  27. 2022Is Power-Seeking AI an Existential Risk?cited by 101
  28. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 95
  29. 2023Evaluating Language-Model Agents on Realistic Autonomous Taskscited by 94
  30. 2023How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questionscited by 82
  31. 2024CogBench: a large language model walks into a psychology labcited by 60
  32. 2024Thousands of AI Authors on the Future of AIcited by 48
  33. 2023Evaluating Superhuman Models with Consistency Checkscited by 47
  34. 2023Visual cognition in multimodal large language modelscited by 38
  35. 2022Prompt Injection: Parameterization of Fixed Inputscited by 35
  36. 2020AI Research Considerations for Human Existential Safety (ARCHES)cited by 33
  37. 2023Towards best practices in AGI safety and governance: A survey of expert opinioncited by 27
  38. 2023Evaluating General-Purpose AI with Psychometricscited by 23

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 7; 2026: 1.

11 papers.

  1. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 57
  2. 2024Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machinacited by 29
  3. 2025Scaling Up Active Testing to Large Language Modelscited by 12
  4. 2024Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecturecited by 10
  5. 2025Paradigms of AI Evaluation: Mapping Goals, Methodologies and Culturecited by 10
  6. 2024Declare and Justify: Explicit assumptions in AI evaluations are necessary for effective regulationcited by 4
  7. 2025What Makes an Evaluation Useful? Common Pitfalls and Best Practicescited by 1
  8. 2025Societal Capacity Assessment Framework: Measuring Resilience to Inform Advanced AI Risk Managementcited by 1
  9. 2025Do Large Language Models Show Biases in Causal Learning? Insights from Contingency Judgmentcited by 1
  10. 2025A Conceptual Framework for AI Capability Evaluationscited by 0
  11. 2026Red Teaming LLMs as Socio-Technical Practice: From Exploration and Data Creation to Evaluationcited by 0