mlsanityData to 7 Oct 2026

AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models

arXiv 2304.06364 ↗First posted 13 Apr 2023cs.CL cs.AIcited by 701

Not on the map. Why?

Cited by 141 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Evaluating the general abilities of foundation models to tackle human-level tasks is a vital aspect of their development and application in the pursuit of Artificial General Intelligence (AGI). Traditional benchmarks, which rely on artificial datasets, may not accurately represent human-level capabilities. In this paper, we introduce AGIEval, a novel benchmark specifically designed to assess foundation model in the context of human-centric standardized exams, such as college entrance exams, law school admission tests, math competitions, and lawyer qualification tests. We evaluate several state-of-the-art foundation models, including GPT-4, ChatGPT, and Text-Davinci-003, using this benchmark. Impressively, GPT-4 surpasses average human performance on SAT, LSAT, and math competitions, attaining a 95% accuracy rate on the SAT Math test and a 92.5% accuracy on the English test of the Chinese national college entrance exam. This demonstrates the extraordinary performance of contemporary foundation models. In contrast, we also find that GPT-4 is less proficient in tasks that require complex reasoning or specific domain knowledge. Our comprehensive analyses of model capabilities (understanding, knowledge, reasoning, and calculation) reveal these models' strengths and limitations, providing valuable insights into future directions for enhancing their general capabilities. By concentrating on tasks pertinent to human cognition and decision-making, our benchmark delivers a more meaningful and robust evaluation of foundation models' performance in real-world scenarios. The data, code, and all model outputs are released in https://github.com/ruixiangcui/AGIEval.

Key facts. Cited by 701 papers in the map (data through 7 Oct 2026). Builds on 28 of its 41 references that are in the map. First posted on arXiv 13 Apr 2023.

Builds on

28 of 41 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
16 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  2. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  4. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  5. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  6. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  7. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  8. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  9. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  10. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  11. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  12. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  13. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  14. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  15. 2019JEC-QA: A Legal-Domain Question Answering Datasetcited by 115
  16. 2021From LSAT: The Progress and Challenges of Complex Reasoningcited by 43

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 124; 2024: 222; 2025: 225; 2026: 130.

701 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Qwen Technical Reportcited by 3,273
  5. 2023Mistral 7Bcited by 3,065
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  8. 2024Mixtral of Expertscited by 1,785
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  10. 2025Gemma 3 Technical Reportcited by 1,686
  11. 2025s1: Simple test-time scalingcited by 1,311
  12. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
38 more
  1. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  3. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  4. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  5. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  6. 2024DeepSeek-V3 Technical Reportcited by 724
  7. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  8. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  9. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  10. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  11. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  12. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  13. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  14. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  15. 2023The Falcon Series of Open Language Modelscited by 526
  16. 2025Kimi K2: Open Agentic Intelligencecited by 523
  17. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  18. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  19. 2025Humanity's Last Examcited by 467
  20. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  21. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  22. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 402
  23. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 381
  24. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  25. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 348
  26. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
  27. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  28. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 283
  29. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
  30. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 261
  31. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  32. 2023Challenges and Applications of Large Language Modelscited by 251
  33. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  34. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 247
  35. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
  36. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  37. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 211
  38. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204

50 most cited of 701 All 701 (JSON)