mlsanityData to 14 Oct 2026

Measuring short-form factuality in large language models

arXiv 2411.04368 ↗First posted 7 Nov 2024cs.CLcited by 373

Not on the map. Why?

Cited by 93 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present SimpleQA, a benchmark that evaluates the ability of language models to answer short, fact-seeking questions. We prioritized two properties in designing this eval. First, SimpleQA is challenging, as it is adversarially collected against GPT-4 responses. Second, responses are easy to grade, because questions are created such that there exists only a single, indisputable answer. Each answer in SimpleQA is graded as either correct, incorrect, or not attempted. A model with ideal behavior would get as many questions correct as possible while not attempting the questions for which it is not confident it knows the correct answer. SimpleQA is a simple, targeted evaluation for whether models "know what they know," and our hope is that this benchmark will remain relevant for the next few generations of frontier models. SimpleQA can be found at https://github.com/openai/simple-evals.

Key facts. Cited by 373 papers in the map (data through 14 Oct 2026). Builds on 10 of its 19 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

10 of 19 references.

References · 10

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  4. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  5. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  6. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generationcited by 162
  7. 2024Long-form factuality in large language modelscited by 155
  8. 2023Do Language Models Know When They're Hallucinating References?cited by 116
  9. 2023FELM: Benchmarking Factuality Evaluation of Large Language Modelscited by 66
  10. 2023Evaluating Hallucinations in Chinese Large Language Modelscited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 194; 2026: 175.

373 papers.

Sort:

By area:

  1. 2024Phi-4 Technical Reportcited by 711
  2. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
  3. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  4. 2025Kimi K2: Open Agentic Intelligencecited by 567
  5. 2025Humanity's Last Examcited by 491
  6. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
  7. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  8. 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 229
  9. 2025ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarizationcited by 141
  10. 2025Deep Research Agents: A Systematic Examination And Roadmapcited by 128
  11. 2025Memento: Fine-tuning LLM Agents without Fine-tuning LLMscited by 127
  12. 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertaintycited by 117
38 more
  1. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 102
  2. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 97
  3. 2025CWM: An Open-Weights LLM for Research on Code Generation with World Modelscited by 93
  4. 2025HalluLens: LLM Hallucination Benchmarkcited by 92
  5. 2025AdaCoT: Pareto-Optimal Adaptive Chain-of-Thought Triggering via Reinforcement Learningcited by 87
  6. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 77
  7. 2025SealQA: Raising the Bar for Reasoning in Search-Augmented Language Modelscited by 75
  8. 2025SimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Modelscited by 66
  9. 2025Are Reasoning Models More Prone to Hallucination?cited by 64
  10. 2025A Survey of Uncertainty Estimation Methods on Large Language Modelscited by 62
  11. 2025Trading Inference-Time Compute for Adversarial Robustnesscited by 61
  12. 2025WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Modelcited by 61
  13. 2025SimpleDeepSearcher: Deep Information Seeking via Web-Powered Reasoning Trajectory Synthesiscited by 55
  14. 2025SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledgecited by 54
  15. 2025The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Inputcited by 52
  16. 2026DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agentscited by 50
  17. 2025Open Deep Search: Democratizing Search with Open-source Reasoning Agentscited by 49
  18. 2025An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agentscited by 46
  19. 2025MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agentscited by 42
  20. 2025Answer Matching Outperforms Multiple Choice for Language Model Evaluationcited by 41
  21. 2025Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in Large Language Modelscited by 40
  22. 2025CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Rewardcited by 40
  23. 2025Reasoning about Uncertainty: Do Reasoning Models Know When They Don't Know?cited by 38
  24. 2025ReportBench: Evaluating Deep Research Agents via Academic Survey Taskscited by 38
  25. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  26. 2025Optimizing Model Selection for Compound AI Systemscited by 36
  27. 2025Predictions as Surrogates: Revisiting Surrogate Outcomes in the Age of AIcited by 34
  28. 2025Search Arena: Analyzing Search-Augmented LLMscited by 34
  29. 2025Beyond GPT-5: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routingcited by 34
  30. 2025DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesiscited by 34
  31. 2025Real-Time Detection of Hallucinated Entities in Long-Form Generationcited by 34
  32. 2026When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationcited by 33
  33. 2025MedBrowseComp: Benchmarking Medical Deep Research and Computer Usecited by 30
  34. 2024Knowledge Graphs, Large Language Models, and Hallucinations: An NLP Perspectivecited by 29
  35. 2025The Mirage of Model Editing: Revisiting Evaluation in the Wildcited by 27
  36. 2025MemSearcher: Training LLMs to Reason, Search and Manage Memory via End-to-End Reinforcement Learningcited by 27
  37. 2025MA-RAG: Multi-Agent Retrieval-Augmented Generation via Collaborative Chain-of-Thought Reasoningcited by 26
  38. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 25

50 most cited of 373 All 373 (JSON)