mlsanityData to 14 Oct 2026

Localizing and Mitigating Errors in Long-form Question Answering

arXiv 2407.11930 ↗First posted 16 Jul 2024cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Long-form question answering (LFQA) aims to provide thorough and in-depth answers to complex questions, enhancing comprehension. However, such detailed responses are prone to hallucinations and factual inconsistencies, challenging their faithful evaluation. This work introduces HaluQuestQA, the first hallucination dataset with localized error annotations for human-written and model-generated LFQA answers. HaluQuestQA comprises 698 QA pairs with 1.8k span-level error annotations for five different error types by expert annotators, along with preference judgments. Using our collected data, we thoroughly analyze the shortcomings of long-form answers and find that they lack comprehensiveness and provide unhelpful references. We train an automatic feedback model on this dataset that predicts error spans with incomplete information and provides associated explanations. Finally, we propose a prompt-based approach, Error-informed refinement, that uses signals from the learned feedback model to refine generated answers, which we show reduces errors and improves answer quality across multiple models. Furthermore, humans find answers generated by our approach comprehensive and highly prefer them (84%) over the baseline answers.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 44 of its 70 references that are in the map. First posted on arXiv 16 Jul 2024.

Builds on

44 of 70 references.

References · 44

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  12. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
32 more
  1. 2023LIMA: Less Is More for Alignmentcited by 1,100
  2. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
  3. 2023Large Language Modelscited by 780
  4. 2019ELI5: Long Form Question Answeringcited by 575
  5. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  6. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 407
  7. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  8. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 338
  9. 2023Evaluating Verifiability in Generative Search Enginescited by 303
  10. 2022Teaching language models to support answers with verified quotescited by 292
  11. 2022Generating Sequences by Learning to Self-Correctcited by 277
  12. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  13. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 234
  14. 2023Fine-tuning Language Models for Factualitycited by 221
  15. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 202
  16. 2021Hurdles to Progress in Long-form Question Answeringcited by 167
  17. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 142
  18. 2020Evaluating Factuality in Generation with Dependency-level Entailmentcited by 122
  19. 2023SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Researchcited by 115
  20. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  21. 2024On the Conversational Persuasiveness of Large Language Models: A Randomized Controlled Trialcited by 109
  22. 2023Generating Benchmarks for Factuality Evaluation of Language Modelscited by 105
  23. 2023Shepherd: A Critic for Language Model Generationcited by 97
  24. 2023Towards Mitigating Hallucination in Large Language Models via Self-Reflectioncited by 81
  25. 2023ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmarkcited by 77
  26. 2024TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarizationcited by 66
  27. 2024Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translationcited by 62
  28. 2022Read before Generate! Faithful Long Form Question Answering with Machine Readingcited by 47
  29. 2023LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedbackcited by 39
  30. 2021Cross-replication Reliability -- An Empirical Approach to Interpreting Inter-rater Reliabilitycited by 31
  31. 2022SNaC: Coherence Error Detection for Narrative Summarizationcited by 19
  32. 2024Self-Consistent Decoding for More Factual Open Responsescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?cited by 10
  2. 2025UCSC at SemEval-2025 Task 3: Context, Models and Prompt Optimization for Automated Hallucination Detection in LLM Outputcited by 0
  3. 2026LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answeringcited by 0
  4. 2026JuICE: A Benchmark for Evaluating LLM-Judge in Identifying Cultural Errorscited by 0