mlsanityData to 7 Oct 2026

A Critical Evaluation of Evaluations for Long-form Question Answering

arXiv 2305.18201 ↗First posted 29 May 2023cs.CLcited by 111

Not on the map. Why?

Abstract (arXiv)

Long-form question answering (LFQA) enables answering a wide range of questions, but its flexibility poses enormous challenges for evaluation. We perform the first targeted study of the evaluation of long-form answers, covering both human and automatic evaluation practices. We hire domain experts in seven areas to provide preference judgments over pairs of answers, along with free-form justifications for their choices. We present a careful analysis of experts' evaluation, which focuses on new aspects such as the comprehensiveness of the answer. Next, we examine automatic text generation metrics, finding that no existing metrics are predictive of human preference judgments. However, some metrics correlate with fine-grained aspects of answers (e.g., coherence). We encourage future work to move away from a single "overall score" of the answer and adopt a multi-faceted evaluation, targeting aspects such as factuality and completeness. We publicly release all of our annotations and code to spur future work into LFQA evaluation.

Key facts. Cited by 111 papers in the map (data through 7 Oct 2026). Builds on 41 of its 69 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

41 of 69 references.

References · 41

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  12. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
29 more
  1. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  2. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 719
  3. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  4. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  5. 2019ELI5: Long Form Question Answeringcited by 567
  6. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  7. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  8. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  9. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  10. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  11. 2020Evaluation of Text Generation: A Surveycited by 275
  12. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  13. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  14. 2021Hurdles to Progress in Long-form Question Answeringcited by 166
  15. 2022Improving Passage Retrieval with Zero-Shot Question Generationcited by 164
  16. 2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamicscited by 159
  17. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  18. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  19. 2020Evaluating Factuality in Generation with Dependency-level Entailmentcited by 122
  20. 2022Evaluating Human-Language Model Interactioncited by 90
  21. 2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
  22. 2021BlonDe: An Automatic Evaluation Metric for Document-level Machine Translationcited by 58
  23. 2022Read before Generate! Faithful Long Form Question Answering with Machine Readingcited by 47
  24. 2020MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metricscited by 42
  25. 2021Bidimensional Leaderboards: Generate and Evaluate Language Hand in Handcited by 32
  26. 2022SNaC: Coherence Error Detection for Narrative Summarizationcited by 19
  27. 2020Adversarial NLI for Factual Correctness in Text Summarisation Modelscited by 13
  28. 2022FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluationcited by 13
  29. 2022Modeling Exemplification in Long-form Question Answering via Retrievalcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 16; 2024: 45; 2025: 35; 2026: 15.

111 papers.

Sort:

By area:

  1. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 800
  2. 2023Large Language Model Alignment: A Surveycited by 241
  3. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  4. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 150
  5. 2024Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Modelscited by 150
  6. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  7. 2024VERISCORE: Evaluating the factuality of verifiable claims in long-form text generationcited by 114
  8. 2023ExpertQA: Expert-Curated Questions and Attributed Answerscited by 96
  9. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoningcited by 96
  10. 2024CRAG -- Comprehensive RAG Benchmarkcited by 94
  11. 2023Human Feedback is not Gold Standardcited by 82
  12. 2024Adaptive Activation Steering: A Tuning-Free LLM Truthfulness Improvement Method for Diverse Hallucinations Categoriescited by 74
38 more
  1. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  2. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 71
  3. 2023Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarizationcited by 69
  4. 2023HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attributioncited by 55
  5. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  6. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 46
  7. 2023Understanding Retrieval Augmentation for Long-Form Question Answeringcited by 45
  8. 2024Into the Unknown Unknowns: Engaged Human Learning through Participation in Language Model Agent Conversationscited by 43
  9. 2024Inverse Constitutional AI: Compressing Preferences into Principlescited by 39
  10. 2025Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistenciescited by 39
  11. 2023Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Modelscited by 35
  12. 2024Suri: Multi-constraint Instruction Following for Long-form Text Generationcited by 34
  13. 2024Direct Judgement Preference Optimizationcited by 33
  14. 2024How Susceptible are LLMs to Influence in Prompts?cited by 32
  15. 2024Long2RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recallcited by 32
  16. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 32
  17. 2024PEDANTS: Cheap but Effective and Interpretable Answer Equivalencecited by 31
  18. 2024CaLMQA: Exploring culturally specific long-form question answering across 23 languagescited by 31
  19. 2024Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignmentcited by 26
  20. 2024Model Internals-based Answer Attribution for Trustworthy Retrieval-Augmented Generationcited by 25
  21. 2025The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Modelscited by 24
  22. 2024PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Modelscited by 23
  23. 2024On the Evaluation of Machine-Generated Reportscited by 23
  24. 2023Assessing Large Language Models on Climate Informationcited by 22
  25. 2025Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settingscited by 22
  26. 2025LLM-powered Multi-agent Framework for Goal-oriented Learning in Intelligent Tutoring Systemcited by 21
  27. 2025PeerQA: A Scientific Question Answering Dataset from Peer Reviewscited by 21
  28. 2024OLAPH: Improving Factuality in Biomedical Long-form Question Answeringcited by 20
  29. 2024Long-Form Answers to Visual Questions from Blind and Low Vision Peoplecited by 18
  30. 2026Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generationcited by 18
  31. 2025BLEUBERI: BLEU is a surprisingly effective reward for instruction followingcited by 17
  32. 2023Reasons to Reject? Aligning Language Models with Judgmentscited by 15
  33. 2025InfoDeepSeek: Benchmarking Agentic Information Seeking for Retrieval-Augmented Generationcited by 15
  34. 2023Pregnant Questions: The Importance of Pragmatic Awareness in Maternal Health Question Answeringcited by 13
  35. 2024Learning to Generate Answers with Citations via Factual Consistency Modelscited by 13
  36. 2025HoH: A Dynamic Benchmark for Evaluating the Impact of Outdated Information on Retrieval-Augmented Generationcited by 13
  37. 2024AdvisorQA: Towards Helpful and Harmless Advice-seeking Question Answering with Collective Intelligencecited by 12
  38. 2024DebateQA: Evaluating Question Answering on Debatable Knowledgecited by 12

50 most cited of 111 All 111 (JSON)