mlsanityData to 7 Oct 2026

SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge

arXiv 2509.07968 ↗First posted 9 Sep 2025cs.CLcited by 44

Not on the map. Why?

Abstract (arXiv)

We introduce SimpleQA Verified, a 1,000-prompt benchmark for evaluating Large Language Model (LLM) short-form factuality based on OpenAI's SimpleQA. It addresses critical limitations in OpenAI's benchmark, including noisy and incorrect labels, topical biases, and question redundancy. SimpleQA Verified was created through a rigorous multi-stage filtering process involving de-duplication, topic balancing, and source reconciliation to produce a more reliable and challenging evaluation set, alongside improvements in the autorater prompt. On this new benchmark, Gemini 2.5 Pro achieves a state-of-the-art F1-score of 55.6, outperforming other frontier models, including GPT-5. This work provides the research community with a higher-fidelity tool to track genuine progress in parametric model factuality and to mitigate hallucinations. The benchmark dataset, evaluation code, and leaderboard are available at: https://www.kaggle.com/benchmarks/deepmind/simpleqa-verified.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 18 of its 25 references that are in the map. First posted on arXiv 9 Sep 2025.

Builds on

18 of 25 references.

References · 18

  1. 2024GPT-4o System Cardcited by 4,467
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  3. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  7. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  8. 2024Measuring short-form factuality in large language modelscited by 345
  9. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 308
  10. 2021Measuring Attribution in Natural Language Generation Modelscited by 253
  11. 2022RealTime QA: What's the Answer Right Now?cited by 239
  12. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generationcited by 158
6 more
  1. 2023Fact-Checking Complex Claims with Program-Guided Reasoningcited by 146
  2. 2024Long-form factuality in large language modelscited by 146
  3. 2024VERISCORE: Evaluating the factuality of verifiable claims in long-form text generationcited by 114
  4. 2024CRAG -- Comprehensive RAG Benchmarkcited by 94
  5. 2023FELM: Benchmarking Factuality Evaluation of Large Language Modelscited by 66
  6. 2025The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Inputcited by 49

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 38.

44 papers.

Sort:

By area:

  1. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  2. 2026DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agentscited by 44
  3. 2026When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationcited by 29
  4. 2026The Confidence Dichotomy: Analyzing and Mitigating Miscalibration in Tool-Use Agentscited by 14
  5. 2026Thinking to Recall: How Reasoning Unlocks Parametric Knowledge in LLMscited by 13
  6. 2026Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?cited by 11
  7. 2026From RAG to Agentic RAG for Faithful Islamic Question Answeringcited by 9
  8. 2026The Truthfulness Spectrum Hypothesiscited by 7
  9. 2026The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibilitycited by 6
  10. 2026Measuring Progress Toward AGI: A Cognitive Frameworkcited by 6
  11. 2025Open Agent Specification (Agent Spec): A Unified Representation for AI Agentscited by 5
  12. 2026On Calibration of Large Language Models: From Response To Capabilitycited by 5
32 more
  1. 2025SafeSearch: Automated Red-Teaming of LLM-Based Search Agentscited by 4
  2. 2026Can Large Language Models Understand, Reason About, and Generate Code-Switched Text?cited by 4
  3. 2026Hallucinations Undermine Trust; Metacognition is a Way Forwardcited by 4
  4. 2026ICA: Information-Aware Credit Assignment for Visually Grounded Long-Horizon Information-Seeking Agentscited by 3
  5. 2026Fanar-Sadiq: A Multi-Agent Architecture for Grounded Islamic QAcited by 3
  6. 2026Faithfulness Metrics Don't Measure Faithfulness: A Meta-Evaluation with Ground Truthcited by 3
  7. 2026Intern-S2-Preview: Scientific Agentic Foundation Modelcited by 3
  8. 2025Comprehensiveness Metrics for Automatic Evaluation of Factual Recall in Text Generationcited by 2
  9. 2025A Coherence-Based Measure of AGIcited by 2
  10. 2026Batch-of-Thought: Cross-Instance Learning for Enhanced LLM Reasoningcited by 2
  11. 2026Mitigating Factual Hallucination in Large Reasoning Models via Mixed-Mode Advantage Regularizationcited by 2
  12. 2026Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigmscited by 2
  13. 2025KoSimpleQA: A Korean Factuality Benchmark with an Analysis of Reasoning LLMscited by 1
  14. 2026Benchmarking Deflection and Hallucination in Large Vision-Language Modelscited by 1
  15. 2026Code-Switching Reveals Language Anchoring in Multilingual LLMscited by 1
  16. 2026The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Contextcited by 1
  17. 2026SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agentscited by 1
  18. 2025HEART: Emotionally-Driven Test-Time Scaling of Language Modelscited by 0
  19. 2026Toward Human-AI Complementarity Across Diverse Taskscited by 0
  20. 2026Task-Aware Calibration: Provably Optimal Decoding in LLMscited by 0
  21. 2026APEX: Automated Prompt Engineering eXpert with Dynamic Data Selectioncited by 0
  22. 2026Are Multilingual Models Actually Improving? Isolating True Cross-Lingual Transfercited by 0
  23. 2026Scaling LLM Knowledge Boundaries via Distribution-Optimized Synthesiscited by 0
  24. 2026Co-LMLM: Continuous-Query Limited Memory Language Modelscited by 0
  25. 2026Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibilitycited by 0
  26. 2026L3Cube-IndicQuest v2: A Large-Scale Multilingual Benchmark for Evaluating Factual Knowledge of Large Language Models Across Indic Languagescited by 0
  27. 2026Why RAGs Hallucinate: Penalty-Aware Evaluation of Retrieval-Augmented Generation Systems with Knowledge-Gap Canariescited by 0
  28. 2026F2DR: A Fine-Grained Full-Pipeline Reward Framework for DeepSearch Workflowscited by 0
  29. 2026RLVR2: Reinforcement Learning with Verifiable Rubric-based Rankingcited by 0
  30. 2026Calibration, Not Answer Selection: Distilling Internal Confidence in Reasoning Modelscited by 0
  31. 2026Beyond Answer Confidence: A Controlled Audit of Self-Knowledge in a Black-Box Decision Modelcited by 0
  32. 2026Generalization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMscited by 0