mlsanityData to 7 Oct 2026

Humanity's Last Exam

arXiv 2501.14249 ↗First posted 24 Jan 2025cs.LG cs.AI cs.CLcited by 467

Not on the map. Why?

Cited by 205 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a significant gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai.

Key facts. Cited by 467 papers in the map (data through 7 Oct 2026). Builds on 51 of its 75 references that are in the map. First posted on arXiv 24 Jan 2025.

Builds on

51 of 75 references.

References · 51

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
38 more
  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  4. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  5. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  6. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  7. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  8. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  10. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  11. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  12. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  13. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  14. 2021Measuring Coding Challenge Competence With APPScited by 879
  15. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  16. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  17. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  18. 2018CoQA: A Conversational Question Answering Challengecited by 674
  19. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  20. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  21. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
  22. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 406
  23. 2024Measuring short-form factuality in large language modelscited by 345
  24. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 277
  25. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 227
  26. 2024Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Modelscited by 156
  27. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 153
  28. 2024LAB-Bench: Measuring Capabilities of Language Models for Biology Researchcited by 138
  29. 2021PixMix: Dreamlike Pictures Comprehensively Improve Safety Measurescited by 131
  30. 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 130
  31. 2024Evaluating Frontier Models for Dangerous Capabilitiescited by 121
  32. 2022Mapping global dynamics of benchmark creation and saturation in artificial intelligencecited by 71
  33. 2024Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligencecited by 65
  34. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agentscited by 61
  35. 2021What Would Jiminy Cricket Do? Towards Agents That Behave Morallycited by 57
  36. 2024How predictable is language model benchmark performance?cited by 36
  37. 2019A BERT Baseline for the Natural Questionscited by 33
  38. 2024Not All LLM Reasoners Are Created Equalcited by 29

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 232; 2026: 235.

467 papers.

Sort:

By area:

  1. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  2. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  3. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 616
  4. 2025Kimi K2: Open Agentic Intelligencecited by 523
  5. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  6. 2025LLMs Get Lost In Multi-Turn Conversationcited by 400
  7. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  8. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 364
  9. 2025WebThinker: Empowering Large Reasoning Models with Deep Research Capabilitycited by 320
  10. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  11. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
  12. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
38 more
  1. 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 212
  2. 2026Gemma 4 Technical Reportcited by 187
  3. 2025WebDancer: Towards Autonomous Information Seeking Agencycited by 180
  4. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  5. 2026MiMo-V2-Flash Technical Reportcited by 163
  6. 2025Scaling Latent Reasoning via Looped Language Modelscited by 145
  7. 2025GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Taskscited by 142
  8. 2025Deep Research Agents: A Systematic Examination And Roadmapcited by 122
  9. 2025Memento: Fine-tuning LLM Agents without Fine-tuning LLMscited by 119
  10. 2025WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agentcited by 114
  11. 2025MedReason: Eliciting Factual Medical Reasoning Steps in LLMs via Knowledge Graphscited by 105
  12. 2025Towards Scientific Intelligence: A Survey of LLM-based Scientific Agentscited by 96
  13. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 92
  14. 2025WebExplorer: Explore and Evolve for Training Long-Horizon Web Agentscited by 86
  15. 2025MemEvolve: Meta-Evolution of Agent Memory Systemscited by 81
  16. 2025ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agentscited by 80
  17. 2025Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RLcited by 75
  18. 2025How Hungry is AI? Benchmarking Energy, Water, and Carbon Footprint of LLM Inferencecited by 73
  19. 2025MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scalingcited by 72
  20. 2025WebResearcher: Unleashing unbounded reasoning capability in Long-Horizon Agentscited by 66
  21. 2025WebSailor-V2: Bridging the Chasm to Proprietary Agents via Synthetic Data and Scalable Reinforcement Learningcited by 59
  22. 2025PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Modelscited by 57
  23. 2025Magistralcited by 57
  24. 2026Memento-Skills: Let Agents Design Agentscited by 56
  25. 2025Training a Scientific Reasoning Model for Chemistrycited by 54
  26. 2025Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Searchcited by 53
  27. 2025Reinforcement Learning from Human Feedbackcited by 53
  28. 2025PolyMath: Evaluating Mathematical Reasoning in Multilingual Contextscited by 52
  29. 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Modelcited by 50
  30. 2025From Web Search towards Agentic Deep Research: Incentivizing Search with Reasoning Agentscited by 49
  31. 2025SciMaster: Towards General-Purpose Scientific AI Agents, Part I. X-Master as Foundation: Can We Lead on Humanity's Last Exam?cited by 45
  32. 2025ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestrationcited by 44
  33. 2026DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agentscited by 44
  34. 2026REDSearcher: A Scalable and Cost-Efficient Framework for Long-Horizon Search Agentscited by 43
  35. 2025TxGemma: Efficient and Agentic LLMs for Therapeuticscited by 40
  36. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
  37. 2025The relationship between reasoning and performance in large language models--o3 (mini) thinks harder, not longercited by 39
  38. 2025CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Rewardcited by 39

50 most cited of 467 All 467 (JSON)