mlsanityData to 7 Oct 2026

GPQA: A Graduate-Level Google-Proof Q&A Benchmark

arXiv 2311.12022 ↗First posted 20 Nov 2023cs.AI cs.CLcited by 2,748

Not on the map. Why?

Cited by 1,004 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present GPQA, a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. We ensure that the questions are high-quality and extremely difficult: experts who have or are pursuing PhDs in the corresponding domains reach 65% accuracy (74% when discounting clear mistakes the experts identified in retrospect), while highly skilled non-expert validators only reach 34% accuracy, despite spending on average over 30 minutes with unrestricted access to the web (i.e., the questions are "Google-proof"). The questions are also difficult for state-of-the-art AI systems, with our strongest GPT-4 based baseline achieving 39% accuracy. If we are to use future AI systems to help us answer very hard questions, for example, when developing new scientific knowledge, we need to develop scalable oversight methods that enable humans to supervise their outputs, which may be difficult even if the supervisors are themselves skilled and knowledgeable. The difficulty of GPQA both for skilled non-experts and frontier AI systems should enable realistic scalable oversight experiments, which we hope can help devise ways for human experts to reliably get truthful information from AI systems that surpass human capabilities.

Key facts. Cited by 2,748 papers in the map (data through 7 Oct 2026). Builds on 33 of its 56 references that are in the map. First posted on arXiv 20 Nov 2023.

Builds on

33 of 56 references.

References · 33

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
21 more
  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  3. 2016Concrete Problems in AI Safetycited by 1,582
  4. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  5. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  6. 2022Red Teaming Language Models with Language Modelscited by 983
  7. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  8. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  9. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  10. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  11. 2018AI safety via debatecited by 384
  12. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  13. 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 211
  14. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  15. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
  16. 2023ExpertQA: Expert-Curated Questions and Attributed Answerscited by 96
  17. 2022SQuALITY: Building a Long-Document Summarization Dataset the Hard Waycited by 50
  18. 2020What do Models Learn from Question Answering Datasets?cited by 45
  19. 2023Debate Helps Supervise Unreliable Expertscited by 39
  20. 2019Controlled Crowdsourcing for High-Quality QA-SRL Annotationcited by 34
  21. 2023A Comprehensive Evaluation of Tool-Assisted Generation Strategiescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 175; 2025: 1175; 2026: 1397.

2,748 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  3. 2024Qwen2.5 Technical Reportcited by 4,713
  4. 2025Qwen3-VL Technical Reportcited by 2,534
  5. 2024Qwen2 Technical Reportcited by 2,076
  6. 2025Gemma 3 Technical Reportcited by 1,686
  7. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  8. 2025s1: Simple test-time scalingcited by 1,311
  9. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  10. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  11. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  12. 2025Large Language Diffusion Modelscited by 872
38 more
  1. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  2. 2024DeepSeek-V3 Technical Reportcited by 724
  3. 2025Qwen2.5-Omni Technical Reportcited by 704
  4. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 616
  5. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 564
  6. 2025Kimi K2: Open Agentic Intelligencecited by 523
  7. 2025Dream 7B: Diffusion Large Language Modelscited by 515
  8. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
  9. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  10. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  11. 2025LIMO: Less is More for Reasoningcited by 488
  12. 2025Humanity's Last Examcited by 467
  13. 2025Qwen3-Omni Technical Reportcited by 451
  14. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 450
  15. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  16. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  17. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 423
  18. 2025Reasoning Models Don't Always Say What They Thinkcited by 388
  19. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 381
  20. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 373
  21. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  22. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 359
  23. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  24. 2025WebThinker: Empowering Large Reasoning Models with Deep Research Capabilitycited by 320
  25. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  26. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  27. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  28. 2025Learning to Reason under Off-Policy Guidancecited by 266
  29. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 261
  30. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 257
  31. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
  32. 2025TTRL: Test-Time Reinforcement Learningcited by 253
  33. 2024TextGrad: Automatic "Differentiation" via Textcited by 250
  34. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  35. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 244
  36. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  37. 2025Dynamic Early Exit in Reasoning Modelscited by 217
  38. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215

50 most cited of 2,748 All 2,748 (JSON)