mlsanityData to 7 Oct 2026

Evaluating Open-QA Evaluation

arXiv 2305.12421 ↗First posted 21 May 2023cs.CL cs.AIcited by 35

Not on the map. Why?

Abstract (arXiv)

This study focuses on the evaluation of the Open Question Answering (Open-QA) task, which can directly estimate the factuality of large language models (LLMs). Current automatic evaluation methods have shown limitations, indicating that human evaluation still remains the most reliable approach. We introduce a new task, Evaluating QA Evaluation (QA-Eval) and the corresponding dataset EVOUNA, designed to assess the accuracy of AI-generated answers in relation to standard answers within Open-QA. Our evaluation of these methods utilizes human-annotated results to measure their performance. Specifically, the work investigates methods that show high correlation with human evaluations, deeming them more reliable. We also discuss the pitfalls of current methods and methods to improve LLM-based evaluators. We believe this new QA-Eval task and corresponding dataset EVOUNA will facilitate the development of more effective automatic evaluation tools and prove valuable for future research in this area. All resources are available at urlhttps://github.com/wangcunxiang/QA-Eval and it is under the Apache-2.0 License.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 31 of its 52 references that are in the map. First posted on arXiv 21 May 2023.

Builds on

31 of 52 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  9. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  10. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  11. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  12. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
19 more
  1. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  2. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  3. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  4. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  5. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  6. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  7. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  8. 2023GPTScore: Evaluate as You Desirecited by 357
  9. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  10. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
  11. 2021Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answeringcited by 178
  12. 2019End-to-End Open-Domain Question Answering with BERTserinicited by 152
  13. 2019Unifying Human and Statistical Evaluation for Natural Language Generationcited by 127
  14. 2021KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answeringcited by 66
  15. 2021Can Generative Pre-trained Language Models Serve as Knowledge Bases for Closed-book QA?cited by 62
  16. 2023Empower Large Language Model to Perform Better on Industrial Domain-Specific Question Answeringcited by 41
  17. 2020SummPip: Unsupervised Multi-Document Summarization with Sentence Graph Compressioncited by 24
  18. 2022Grape: Knowledge Graph Enhanced Passage Reader for Open-domain Question Answeringcited by 22
  19. 2022A Simple but Effective Pluggable Entity Lookup Table for Pre-trained Language Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 16; 2025: 14; 2026: 3.

35 papers.

Sort:

By area:

  1. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  2. 2024RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generationcited by 59
  3. 2024Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMscited by 55
  4. 2024Can AI Assistants Know What They Don't Know?cited by 47
  5. 2024HelloBench: Evaluating Long Text Generation Capabilities of Large Language Modelscited by 42
  6. 2023Evaluating Hallucinations in Chinese Large Language Modelscited by 33
  7. 2024NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokenscited by 31
  8. 2024Cofca: A Step-Wise Counterfactual Multi-hop QA benchmarkcited by 26
  9. 2025SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reductioncited by 24
  10. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 22
  11. 2024Enhancing Visual Question Answering through Question-Driven Image Captions as Promptscited by 18
  12. 2024xFinder: Large Language Models as Automated Evaluators for Reliable Evaluationcited by 18
23 more
  1. 2024MetaMetrics: Calibrating Metrics For Generation Tasks Using Human Preferencescited by 15
  2. 2024PertEval: Unveiling Real Knowledge Capacity of LLMs with Knowledge-Invariant Perturbationscited by 13
  3. 2025Taxonomy-Aware Evaluation of Vision-Language Modelscited by 11
  4. 2025Diagnosing and Addressing Pitfalls in KG-RAG Datasets: Toward More Reliable Benchmarkingcited by 11
  5. 2024Evaluating the Factuality of Large Language Models using Large-Scale Knowledge Graphscited by 10
  6. 2024A review of faithfulness metrics for hallucination assessment in Large Language Modelscited by 9
  7. 2025ScIRGen: Synthesize Realistic and Large-Scale RAG Dataset for Scientific Researchcited by 7
  8. 2025Cross-Layer Attention Probing for Fine-Grained Hallucination Detectioncited by 4
  9. 2025Investigating Retrieval-Augmented Generation in Quranic Studies: A Study of 13 Open-Source Large Language Modelscited by 2
  10. 2025How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAGcited by 2
  11. 2024Towards Robust Evaluation: A Comprehensive Taxonomy of Datasets and Metrics for Open Domain Question Answering in the Era of Large Language Modelscited by 1
  12. 2024Evaluating the Correctness of Inference Patterns Used by LLMs for Judgmentcited by 1
  13. 2024PPLqa: An Unsupervised Information-Theoretic Quality Metric for Comparing Generative Large Language Modelscited by 1
  14. 2025AirQA: A Comprehensive QA Dataset for AI Research with Instance-Level Evaluationcited by 1
  15. 2025Knowledge-Grounded Agentic Large Language Models for Multi-Hazard Understanding from Reconnaissance Reportscited by 1
  16. 2025SMILE: A Composite Lexical-Semantic Metric for Question-Answering Evaluationcited by 1
  17. 2026The Powerless Noise: How Experimental Settings Shape the Reported Power of Noisecited by 1
  18. 2024'Evaluation des capacit'es de r'eponse de larges mod`eles de langage (LLM) pour des questions d'historienscited by 0
  19. 2025Can LLMs Ask Good Questions?cited by 0
  20. 2025HIVMedQA: Benchmarking large language models for HIV medical decision supportcited by 0
  21. 2025Faithfulness metric fusion: Improving the evaluation of LLM trustworthiness across domainscited by 0
  22. 2026PBSBench: A Multi-Level Vision-Language Framework and Benchmark for Hematopathology Whole Slide Image Interpretationcited by 0
  23. 2026ECUASn: A family of metrics for principled evaluation of uncertainty-augmented systemscited by 0