mlsanityData to 7 Oct 2026

Hurdles to Progress in Long-form Question Answering

arXiv 2103.06332 ↗First posted 10 Mar 2021cs.CL cs.LGcited by 166

Not on the map. Why?

Cited by 25 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The task of long-form question answering (LFQA) involves retrieving documents relevant to a given question and using them to generate a paragraph-length answer. While many models have recently been proposed for LFQA, we show in this paper that the task formulation raises fundamental challenges regarding evaluation and dataset creation that currently preclude meaningful modeling progress. To demonstrate these challenges, we first design a new system that relies on sparse attention and contrastive retriever learning to achieve state-of-the-art performance on the ELI5 LFQA dataset. While our system tops the public leaderboard, a detailed analysis reveals several troubling trends: (1) our system's generated answers are not actually grounded in the documents that it retrieves; (2) ELI5 contains significant train / validation overlap, as at least 81% of ELI5 validation questions occur in paraphrased form in the training set; (3) ROUGE-L is not an informative metric of generated answer quality and can be easily gamed; and (4) human evaluations used for other text generation tasks are unreliable for LFQA. We offer suggestions to mitigate each of these issues, which we hope will lead to more rigorous LFQA research and meaningful progress in the future.

Key facts. Cited by 166 papers in the map (data through 7 Oct 2026). Builds on 42 of its 52 references that are in the map. First posted on arXiv 10 Mar 2021.

Builds on

42 of 52 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  12. 2019Language Models as Knowledge Bases?cited by 1,922
30 more
  1. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  2. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  3. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  4. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  5. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  6. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  7. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  8. 2019ELI5: Long Form Question Answeringcited by 567
  9. 2019Defending Against Neural Fake Newscited by 553
  10. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  11. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  12. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  13. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  14. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  15. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  16. 2020Evaluation of Text Generation: A Surveycited by 275
  17. 2017Challenges in Data-to-Document Generationcited by 226
  18. 2019TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agentscited by 188
  19. 2020Detecting Hallucinated Content in Conditional Neural Sequence Generationcited by 151
  20. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 137
  21. 2019Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reportscited by 96
  22. 2020Pre-training via Paraphrasingcited by 92
  23. 2018How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarkscited by 86
  24. 2018Tensor2Tensor for Neural Machine Translationcited by 85
  25. 2018Conversational AI: The Science Behind the Alexa Prizecited by 63
  26. 2019Sticking to the Facts: Confident Decoding for Faithful Data-to-Text Generationcited by 62
  27. 2020STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generationcited by 56
  28. 2020RoFT: A Tool for Evaluating Human Detection of Machine-Generated Textcited by 36
  29. 2019EvalAI: Towards Better Evaluation Systems for AI Agentscited by 34
  30. 2020On-The-Fly Information Retrieval Augmentation for Language Modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 26; 2023: 39; 2024: 49; 2025: 26; 2026: 8.

166 papers.

Sort:

By area:

  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  2. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  3. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  4. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  5. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  6. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  7. 2023Active Retrieval Augmented Generationcited by 506
  8. 2023Augmented Language Models: a Surveycited by 398
  9. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
  10. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 286
  11. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 231
  12. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
38 more
  1. 2021FeTaQA: Free-form Table Question Answeringcited by 174
  2. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  3. 2023Conformal Language Modelingcited by 163
  4. 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 144
  5. 2022Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidencecited by 142
  6. 2023Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigationcited by 123
  7. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  8. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  9. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  10. 2022Capturing Failures of Large Language Models via Human Cognitive Biasescited by 97
  11. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  12. 2023ExpertQA: Expert-Curated Questions and Attributed Answerscited by 96
  13. 2023WebCPM: Interactive Web Search for Chinese Long-form Question Answeringcited by 93
  14. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  15. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  16. 2023ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agentcited by 78
  17. 2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
  18. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  19. 2022CREPE: Open-Domain Question Answering with False Presuppositionscited by 70
  20. 2023PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documentscited by 61
  21. 2021The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpuscited by 60
  22. 2021GooAQ: Open Question Answering with Diverse Answer Typescited by 53
  23. 2023PURR: Efficiently Editing Language Model Hallucinations by Denoising Language Model Corruptionscited by 53
  24. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  25. 2022SQuALITY: Building a Long-Document Summarization Dataset the Hard Waycited by 50
  26. 2022Read before Generate! Faithful Long Form Question Answering with Machine Readingcited by 47
  27. 2023Won't Get Fooled Again: Answering Questions with False Premisescited by 47
  28. 2023Understanding Retrieval Augmentation for Long-Form Question Answeringcited by 45
  29. 2025ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubricscited by 44
  30. 2024RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answeringcited by 43
  31. 2022A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methodscited by 42
  32. 2023Tree of Clarifications: Answering Ambiguous Questions with Retrieval-Augmented Large Language Modelscited by 42
  33. 2023Empower Large Language Model to Perform Better on Industrial Domain-Specific Question Answeringcited by 41
  34. 2021Evidentiality-guided Generation for Knowledge-Intensive NLP Taskscited by 40
  35. 2021It's not Rocket Science : Interpreting Figurative Language in Narrativescited by 39
  36. 2021FRUIT: Faithfully Reflecting Updated Information in Textcited by 39
  37. 2021What's in a Summary? Laying the Groundwork for Advances in Hospital-Course Summarizationcited by 35
  38. 2023Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Modelscited by 35

50 most cited of 166 All 166 (JSON)