mlsanityData to 14 Oct 2026

Evaluating Open-Domain Question Answering in the Era of Large Language Models

arXiv 2305.06984 ↗First posted 11 May 2023cs.CLcited by 111

Not on the map. Why?

Cited by 20 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Lexical matching remains the de facto evaluation method for open-domain question answering (QA). Unfortunately, lexical matching fails completely when a plausible candidate answer does not appear in the list of gold answers, which is increasingly the case as we shift from extractive to generative models. The recent success of large language models (LLMs) for QA aggravates lexical matching failures since candidate answers become longer, thereby making matching with the gold answers even more challenging. Without accurate evaluation, the true progress in open-domain QA remains unknown. In this paper, we conduct a thorough analysis of various open-domain QA models, including LLMs, by manually evaluating their answers on a subset of NQ-open, a popular benchmark. Our assessments reveal that while the true performance of all models is significantly underestimated, the performance of the InstructGPT (zero-shot) LLM increases by nearly +60%, making it on par with existing top models, and the InstructGPT (few-shot) model actually achieves a new state-of-the-art on NQ-open. We also find that more than 50% of lexical matching failures are attributed to semantically equivalent answers. We further demonstrate that regex matching ranks QA models consistent with human judgments, although still suffering from unnecessary strictness. Finally, we demonstrate that automated evaluation models are a reasonable surrogate for lexical matching in some circumstances, but not for long-form answers generated by LLMs. The automated models struggle in detecting hallucinations in LLM answers and are thus unable to evaluate LLMs. At this time, there appears to be no substitute for human evaluation.

Key facts. Cited by 111 papers in the map (data through 14 Oct 2026). Builds on 39 of its 51 references that are in the map. First posted on arXiv 11 May 2023.

Builds on

39 of 51 references.

References · 39

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  11. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  12. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
27 more
  1. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  2. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  3. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 849
  4. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  5. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  6. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  7. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
  8. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
  9. 2021Measuring Attribution in Natural Language Generation Modelscited by 256
  10. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  11. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  12. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 176
  13. 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoningcited by 169
  14. 2019Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answeringcited by 147
  15. 2022On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?cited by 138
  16. 2021End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answeringcited by 133
  17. 2017Simple and Effective Multi-Paragraph Reading Comprehensioncited by 120
  18. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  19. 2019Reasoning Over Paragraph Effects in Situationscited by 79
  20. 2020Relevance-guided Supervision for OpenQA with ColBERTcited by 71
  21. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 67
  22. 2021NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learnedcited by 52
  23. 2021Semantic Answer Similarity for Evaluating Question Answering Modelscited by 47
  24. 2020MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metricscited by 42
  25. 2021Evidentiality-guided Generation for Knowledge-Intensive NLP Taskscited by 40
  26. 2021R2-D2: A Modular Baseline for Open-Domain Question Answeringcited by 37
  27. 2021What's in a Name? Answer Equivalence For Open-Domain Question Answeringcited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 40; 2025: 43; 2026: 18.

111 papers.

Sort:

By area:

  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  2. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 151
  3. 2023Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentationcited by 147
  4. 2024RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculturecited by 81
  5. 2024Speculative RAG: Enhancing Retrieval Augmented Generation through Draftingcited by 77
  6. 2024SPIQA: A Dataset for Multimodal Question Answering on Scientific Paperscited by 75
  7. 2024SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMscited by 67
  8. 2023Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Modelscited by 66
  9. 2024Understanding LLMs: A Comprehensive Overview from Training to Inferencecited by 59
  10. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferencescited by 58
  11. 2023Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4cited by 48
  12. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 47
38 more
  1. 2025TimeXL: Explainable Multi-modal Time Series Prediction with LLM-in-the-Loopcited by 39
  2. 2025Benchmarking Prompt Sensitivity in Large Language Modelscited by 38
  3. 2023Improving Automatic VQA Evaluation Using Large Language Modelscited by 37
  4. 2023FunQA: Towards Surprising Video Comprehensioncited by 35
  5. 2024VideoCoT: A Video Chain-of-Thought Dataset with Active Annotation Toolcited by 35
  6. 2024Are We in the AI-Generated Text World Already? Quantifying and Monitoring AIGT on Social Mediacited by 35
  7. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  8. 2025Optimizing Temperature for Language Models with Multi-Sample Inferencecited by 33
  9. 2024Few-shot Personalization of LLMs with Mis-aligned Responsescited by 32
  10. 2025Defending against Indirect Prompt Injection by Instruction Detectioncited by 32
  11. 2024D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Modelscited by 31
  12. 2025CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoningcited by 28
  13. 2024Stratified Prediction-Powered Inference for Hybrid Language Model Evaluationcited by 26
  14. 2024Are Large Language Models Reliable Argument Quality Annotators?cited by 24
  15. 2024Multi-Agent Large Language Models for Conversational Task-Solvingcited by 24
  16. 2025Mask-DPO: Generalizable Fine-grained Factuality Alignment of LLMscited by 24
  17. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 20
  18. 2024LoCoCo: Dropping In Convolutions for Long Context Compressioncited by 19
  19. 2024A Survey of Sustainability in Large Language Models: Applications, Economics, and Challengescited by 18
  20. 2024Pirates of the RAG: Adaptively Attacking LLMs to Leak Knowledge Basescited by 18
  21. 2025TableMaster: A Recipe to Advance Table Understanding with Language Modelscited by 18
  22. 2025Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queriescited by 16
  23. 2024Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspectivecited by 15
  24. 2024Are LLMs Correctly Integrated into Software Systems?cited by 15
  25. 2024Logical Consistency of Large Language Models in Fact-checkingcited by 15
  26. 2024ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Modelscited by 14
  27. 2026SmartSearch: Process Reward-Guided Query Refinement for Search Agentscited by 14
  28. 2026CM2: Reinforcement Learning with Checklist Rewards for Multi-Turn and Multi-Step Agentic Tool Usecited by 14
  29. 2023Leveraging LLMs in Scholarly Knowledge Graph Question Answeringcited by 13
  30. 2024Beyond Training: Dynamic Token Merging for Zero-Shot Video Understandingcited by 13
  31. 2024What Makes a Maze Look Like a Maze?cited by 11
  32. 2024Guiding Vision-Language Model Selection for Visual Question-Answering Across Tasks, Domains, and Knowledge Typescited by 10
  33. 2025OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inferencecited by 10
  34. 2024A review of faithfulness metrics for hallucination assessment in Large Language Modelscited by 9
  35. 2024Bayesian Prediction-Powered Inferencecited by 8
  36. 2024Toward Robust RALMs: Revealing the Impact of Imperfect Retrieval on Retrieval-Augmented Language Modelscited by 8
  37. 2024RAG-based Question Answering over Heterogeneous Data and Textcited by 7
  38. 2024A Survey on Multi-modal Machine Translation: Tasks, Methods and Challengescited by 6

50 most cited of 111 All 111 (JSON)