# Read before Generate! Faithful Long Form Question Answering with Machine Reading

arXiv: https://arxiv.org/abs/2203.00343 · HTML: https://mlsanity.com/paper/2203.00343/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 66 references that are in the map.
- First posted on arXiv 1 Mar 2022.

## Builds on (most cited first, 30 of 36)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [ELI5: Long Form Question Answering](https://mlsanity.com/paper/1907.09190/) (2019)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [Wizard of Wikipedia: Knowledge-Powered Conversational agents](https://mlsanity.com/paper/1811.01241/) (2018)
- [Zero-Shot Relation Extraction via Reading Comprehension](https://mlsanity.com/paper/1706.04115/) (2017)
- [Asking and Answering Questions to Evaluate the Factual Consistency of Summaries](https://mlsanity.com/paper/2004.04228/) (2020)
- [NewsQA: A Machine Comprehension Dataset](https://mlsanity.com/paper/1611.09830/) (2016)
- [FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization](https://mlsanity.com/paper/2005.03754/) (2020)
- [SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine](https://mlsanity.com/paper/1704.05179/) (2017)
- [MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension](https://mlsanity.com/paper/1910.09753/) (2019)
- [How Context Affects Language Models' Factual Predictions](https://mlsanity.com/paper/2005.04611/) (2020)
- [Hurdles to Progress in Long-form Question Answering](https://mlsanity.com/paper/2103.06332/) (2021)
- [A Dataset for Document Grounded Conversations](https://mlsanity.com/paper/1809.07358/) (2018)
- [GooAQ: Open Question Answering with Diverse Answer Types](https://mlsanity.com/paper/2104.08727/) (2021)

## Built on by (most cited first, 30 of 47)

- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs](https://mlsanity.com/paper/2406.15927/) (2024)
- [Factuality Enhanced Language Models for Open-Ended Text Generation](https://mlsanity.com/paper/2206.04624/) (2022)
- [Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation](https://mlsanity.com/paper/2305.15852/) (2023)
- [A Critical Evaluation of Evaluations for Long-form Question Answering](https://mlsanity.com/paper/2305.18201/) (2023)
- [Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus](https://mlsanity.com/paper/2311.13230/) (2023)
- [Towards Mitigating Hallucination in Large Language Models via Self-Reflection](https://mlsanity.com/paper/2310.06271/) (2023)
- [Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training](https://mlsanity.com/paper/2210.07688/) (2022)
- [SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs](https://mlsanity.com/paper/2404.13081/) (2024)
- [PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documents](https://mlsanity.com/paper/2305.14564/) (2023)
- [Why Does ChatGPT Fall Short in Providing Truthful Answers?](https://mlsanity.com/paper/2304.10513/) (2023)
- [DelucionQA: Detecting Hallucinations in Domain-specific Question Answering](https://mlsanity.com/paper/2312.05200/) (2023)
- ["According to ...": Prompting Language Models Improves Quoting from Pre-Training Data](https://mlsanity.com/paper/2305.13252/) (2023)
- [Empower Large Language Model to Perform Better on Industrial Domain-Specific Question Answering](https://mlsanity.com/paper/2305.11541/) (2023)
- [Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Models](https://mlsanity.com/paper/2407.08039/) (2024)
- [Estimating the Hallucination Rate of Generative AI](https://mlsanity.com/paper/2406.07457/) (2024)
- [How Do We Answer Complex Questions: Discourse Structure of Long-form Answers](https://mlsanity.com/paper/2203.11048/) (2022)
- [Know When To Stop: A Study of Semantic Drift in Text Generation](https://mlsanity.com/paper/2404.05411/) (2024)
- [OmniThink: Expanding Knowledge Boundaries in Machine Writing through Thinking](https://mlsanity.com/paper/2501.09751/) (2025)
- [KnowTuning: Knowledge-aware Fine-tuning for Large Language Models](https://mlsanity.com/paper/2402.11176/) (2024)
- [Optimizing Retrieval-augmented Reader Models via Token Elimination](https://mlsanity.com/paper/2310.13682/) (2023)
- [Context Generation Improves Open Domain Question Answering](https://mlsanity.com/paper/2210.06349/) (2022)
- [Groundedness in Retrieval-augmented Long-form Generation: An Empirical Study](https://mlsanity.com/paper/2404.07060/) (2024)
- [Integrating Planning into Single-Turn Long-Form Text Generation](https://mlsanity.com/paper/2410.06203/) (2024)
- [Trustworthy Text-to-Image Diffusion Models: A Timely and Focused Survey](https://mlsanity.com/paper/2409.18214/) (2024)
- [SELU: Self-Learning Embodied MLLMs in Unknown Environments](https://mlsanity.com/paper/2410.03303/) (2024)
- [Long-form Question Answering: An Iterative Planning-Retrieval-Generation Approach](https://mlsanity.com/paper/2311.09383/) (2023)
- [Localizing and Mitigating Errors in Long-form Question Answering](https://mlsanity.com/paper/2407.11930/) (2024)
- [A Benchmark Dataset with Larger Context for Non-Factoid Question Answering over Islamic Text](https://mlsanity.com/paper/2409.09844/) (2024)
- [Learning from Many Voices: Literary MT Using Multi-Reference Human and Synthetic Data](https://mlsanity.com/paper/2412.18707/) (2024)

## Abstract (arXiv)

Long-form question answering (LFQA) aims to generate a paragraph-length answer for a given question. While current work on LFQA using large pre-trained model for generation are effective at producing fluent and somewhat relevant content, one primary challenge lies in how to generate a faithful answer that has less hallucinated content. We propose a new end-to-end framework that jointly models answer generation and machine reading. The key idea is to augment the generation model with fine-grained, answer-related salient information which can be viewed as an emphasis on faithful facts. State-of-the-art results on two LFQA datasets, ELI5 and MS MARCO, demonstrate the effectiveness of our method, in comparison with strong baselines on automatic and human evaluation metrics. A detailed analysis further proves the competency of our methods in generating fluent, relevant, and more faithful answers.