mlsanityData to 14 Oct 2026

ExpertQA: Expert-Curated Questions and Attributed Answers

arXiv 2309.07852 ↗First posted 14 Sep 2023cs.CL cs.AIcited by 97

Not on the map. Why?

Abstract (arXiv)

As language models are adopted by a more sophisticated and diverse set of users, the importance of guaranteeing that they provide factually correct information supported by verifiable sources is critical across fields of study. This is especially the case for high-stakes fields, such as medicine and law, where the risk of propagating false information is high and can lead to undesirable societal consequences. Previous work studying attribution and factuality has not focused on analyzing these characteristics of language model outputs in domain-specific scenarios. In this work, we conduct human evaluation of responses from a few representative systems along various axes of attribution and factuality, by bringing domain experts in the loop. Specifically, we collect expert-curated questions from 484 participants across 32 fields of study, and then ask the same experts to evaluate generated responses to their own questions. In addition, we ask experts to improve upon responses from language models. The output of our analysis is ExpertQA, a high-quality long-form QA dataset with 2177 questions spanning 32 fields, along with verified answers and attributions for claims in the answers.

Key facts. Cited by 97 papers in the map (data through 14 Oct 2026). Builds on 59 of its 90 references that are in the map. First posted on arXiv 14 Sep 2023.

Builds on

59 of 90 references.

References · 59

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  8. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  9. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  11. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  12. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
38 more
  1. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  2. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  3. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  4. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  5. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  6. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
  7. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  8. 2018CoQA: A Conversational Question Answering Challengecited by 683
  9. 2023The Internal State of an LLM Knows When It's Lyingcited by 659
  10. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
  11. 2019ELI5: Long Form Question Answeringcited by 575
  12. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 543
  13. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 378
  14. 2021A Survey on Automated Fact-Checkingcited by 378
  15. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 336
  16. 2023Evaluating Verifiability in Generative Search Enginescited by 303
  17. 2022Teaching language models to support answers with verified quotescited by 292
  18. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 287
  19. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 287
  20. 2021Measuring Attribution in Natural Language Generation Modelscited by 256
  21. 2022Transformer Memory as a Differentiable Search Indexcited by 247
  22. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 236
  23. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 234
  24. 2021Hurdles to Progress in Long-form Question Answeringcited by 167
  25. 2022Rethinking with Retrieval: Faithful Large Language Model Inferencecited by 145
  26. 2021A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generationcited by 138
  27. 2018emrQA: A Large Corpus for Question Answering on Electronic Medical Recordscited by 135
  28. 2023WiCE: Real-World Entailment for Claims in Wikipediacited by 120
  29. 2023Do Language Models Know When They're Hallucinating References?cited by 116
  30. 2021Truthful AI: Developing and governing AI that does not liecited by 113
  31. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  32. 2023Complex Claim Verification with Evidence Retrieved in the Wildcited by 110
  33. 2018The Fact Extraction and VERification (FEVER) Shared Taskcited by 108
  34. 2023Generating Benchmarks for Factuality Evaluation of Language Modelscited by 105
  35. 2021Decontextualization: Making Sentences Stand-Alonecited by 102
  36. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  37. 2021Evaluating Attribution in Dialogue Systems: The BEGIN Benchmarkcited by 77
  38. 2023Automatic Evaluation of Attribution by Large Language Modelscited by 72

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 34; 2025: 27; 2026: 28.

97 papers.

Sort:

By area:

  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  3. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  4. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  5. 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 54
  6. 2024Evaluating LLMs at Detecting Errors in LLM Responsescited by 50
  7. 2025ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubricscited by 49
  8. 2024CiteME: Can Language Models Accurately Cite Scientific Claims?cited by 48
  9. 2024FLAME: Factuality-Aware Alignment for Large Language Modelscited by 47
  10. 2023Understanding Retrieval Augmentation for Long-Form Question Answeringcited by 45
  11. 2024HelloBench: Evaluating Long Text Generation Capabilities of Large Language Modelscited by 44
  12. 2025GFM-RAG: Graph Foundation Model for Retrieval Augmented Generationcited by 40
38 more
  1. 2024RAR-b: Reasoning as Retrieval Benchmarkcited by 36
  2. 2025KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Modelcited by 36
  3. 2023HealthFC: Verifying Health Claims with Evidence-Based Medical Fact-Checkingcited by 35
  4. 2024Ragnar"ok: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Trackcited by 32
  5. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  6. 2023The Trickle-down Impact of Reward (In-)consistency on RLHFcited by 29
  7. 2024Fact-and-Reflection (FaR) Improves Confidence Calibration of Large Language Modelscited by 27
  8. 2024AttributionBench: How Hard is Automatic Attribution Evaluation?cited by 26
  9. 2024CLAPNQ: Cohesive Long-form Answers from Passages in Natural Questions for RAG systemscited by 25
  10. 2024Nearest Neighbor Speculative Decoding for LLM Generation and Attributioncited by 25
  11. 2024MEDIC: Comprehensive Evaluation of Leading Indicators for LLM Safety and Utility in Clinical Applicationscited by 25
  12. 2024FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluationcited by 25
  13. 2024Claim Verification in the Age of Large Language Models: A Surveycited by 24
  14. 2024How well do LLMs cite relevant medical references? An evaluation framework and analysescited by 21
  15. 2024Ground Every Sentence: Improving Retrieval-Augmented LLMs with Interleaved Reference-Claim Generationcited by 21
  16. 2024Atomic Fact Decomposition Helps Attributed Question Answeringcited by 21
  17. 2025SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Taskscited by 20
  18. 2025CiteEval: Principle-Driven Citation Evaluation for Source Attributioncited by 19
  19. 2024Core: Robust Factual Precision with Informative Sub-Claim Identificationcited by 16
  20. 2024CitaLaw: Enhancing LLM with Citations in Legal Domaincited by 16
  21. 2024Verifiable by Design: Aligning Language Models to Quote from Pre-Training Datacited by 13
  22. 2025Towards Omni-RAG: Comprehensive Retrieval-Augmented Generation for Large Language Models in Medical Applicationscited by 13
  23. 2025Verifying the Verifiers: Unveiling Pitfalls and Potentials in Fact Verifierscited by 13
  24. 2024Graphusion: A RAG Framework for Knowledge Graph Construction with a Global Perspectivecited by 12
  25. 2024Contextualized Evaluations: Judging Language Model Responses to Underspecified Queriescited by 12
  26. 2024Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluationcited by 11
  27. 2025EvalAgent: Discovering Implicit Evaluation Criteria from the Webcited by 11
  28. 2023Integrating UMLS Knowledge into Large Language Models for Medical Question Answeringcited by 10
  29. 2025CoCoA: Confidence and Context-Aware Adaptive Decoding for Resolving Knowledge Conflicts in Large Language Modelscited by 10
  30. 2024A review of faithfulness metrics for hallucination assessment in Large Language Modelscited by 9
  31. 2025EnronQA: Towards Personalized RAG over Private Documentscited by 9
  32. 2023Generalization Analogies: A Testbed for Generalizing AI Oversight to Hard-To-Measure Domainscited by 8
  33. 2024DOLOMITES: Domain-Specific Long-Form Methodical Taskscited by 8
  34. 2025FaStfact: Faster, Stronger Long-Form Factuality Evaluations in LLMscited by 8
  35. 2025Attribution, Citation, and Quotation: A Survey of Evidence-based Text Generation with Large Language Modelscited by 7
  36. 2025Reasoning-CV: Fine-tuning Powerful Reasoning LLMs for Knowledge-Assisted Claim Verificationcited by 6
  37. 2026Detecting and Correcting Reference Hallucinations in Commercial LLMs and Deep Research Agentscited by 6
  38. 2025Building Benchmarks from the Ground Up: Community-Centered Evaluation of LLMs in Healthcare Chatbot Settingscited by 5

50 most cited of 97 All 97 (JSON)