# Focused Attention Improves Document-Grounded Generation

arXiv: https://arxiv.org/abs/2104.12714 · HTML: https://mlsanity.com/paper/2104.12714/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 58 references that are in the map.
- First posted on arXiv 26 Apr 2021.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [Wizard of Wikipedia: Knowledge-Powered Conversational agents](https://mlsanity.com/paper/1811.01241/) (2018)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset](https://mlsanity.com/paper/1909.05855/) (2019)
- [A Neural Attention Model for Abstractive Sentence Summarization](https://mlsanity.com/paper/1509.00685/) (2015)
- [Generating Wikipedia by Summarizing Long Sequences](https://mlsanity.com/paper/1801.10198/) (2018)
- [Plan-And-Write: Towards Better Automatic Storytelling](https://mlsanity.com/paper/1811.05701/) (2018)
- [Challenges in Data-to-Document Generation](https://mlsanity.com/paper/1707.08052/) (2017)
- [A Neural Conversational Model](https://mlsanity.com/paper/1506.05869/) (2015)
- [The Second Conversational Intelligence Challenge (ConvAI2)](https://mlsanity.com/paper/1902.00098/) (2019)
- [Visual Storytelling](https://mlsanity.com/paper/1604.03968/) (2016)
- [A Knowledge-Grounded Neural Conversation Model](https://mlsanity.com/paper/1702.01932/) (2017)
- [Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems](https://mlsanity.com/paper/1508.01745/) (2015)
- [Knowledge-Grounded Dialogue Generation with Pre-trained Language Models](https://mlsanity.com/paper/2010.08824/) (2020)
- [A Dataset for Document Grounded Conversations](https://mlsanity.com/paper/1809.07358/) (2018)
- [Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation](https://mlsanity.com/paper/1706.09799/) (2017)
- [Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation](https://mlsanity.com/paper/1701.08251/) (2017)
- [Polite Dialogue Generation Without Parallel Data](https://mlsanity.com/paper/1805.03162/) (2018)
- [Exploring Controllable Text Generation Techniques](https://mlsanity.com/paper/2005.01822/) (2020)
- [Low-Resource Knowledge-Grounded Dialogue Generation](https://mlsanity.com/paper/2002.10348/) (2020)

## Built on by (most cited first, 24 of 24)

- [On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?](https://mlsanity.com/paper/2204.07931/) (2022)
- [FaithDial: A Faithful Benchmark for Information-Seeking Dialogue](https://mlsanity.com/paper/2204.10757/) (2022)
- [Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark](https://mlsanity.com/paper/2105.00071/) (2021)
- [OpenCQA: Open-ended Question Answering with Charts](https://mlsanity.com/paper/2210.06628/) (2022)
- [Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuning](https://mlsanity.com/paper/2305.15065/) (2023)
- [Am I Me or You? State-of-the-Art Dialogue Models Cannot Maintain an Identity](https://mlsanity.com/paper/2112.05843/) (2021)
- [Multi-Stage Prompting for Knowledgeable Dialogue Generation](https://mlsanity.com/paper/2203.08745/) (2022)
- [Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challenges](https://mlsanity.com/paper/2405.15604/) (2024)
- [Pointwise Mutual Information Based Metric and Decoding Strategy for Faithful Generation in Document Grounded Dialogs](https://mlsanity.com/paper/2305.12191/) (2023)
- [Zemi: Learning Zero-Shot Semi-Parametric Language Models from Multiple Tasks](https://mlsanity.com/paper/2210.00185/) (2022)
- [Evaluating Evaluation Metrics -- The Mirage of Hallucination Detection](https://mlsanity.com/paper/2504.18114/) (2025)
- [KnowPrefix-Tuning: A Two-Stage Prefix-Tuning Framework for Knowledge-Grounded Dialogue Generation](https://mlsanity.com/paper/2306.15430/) (2023)
- [Learning as Conversation: Dialogue Systems Reinforced for Information Acquisition](https://mlsanity.com/paper/2205.14748/) (2022)
- [CorefDiffs: Co-referential and Differential Knowledge Flow in Document Grounded Conversations](https://mlsanity.com/paper/2210.02223/) (2022)
- [Achieving Conversational Goals with Unsupervised Post-hoc Knowledge Injection](https://mlsanity.com/paper/2203.11399/) (2022)
- [Controllable Factuality in Document-Grounded Dialog Systems Using a Noisy Channel Model](https://mlsanity.com/paper/2210.17418/) (2022)
- [Transformers Utilization in Chart Understanding: A Review of Recent Advances & Future Trends](https://mlsanity.com/paper/2410.13883/) (2024)
- [Building Knowledge-Grounded Dialogue Systems with Graph-Based Semantic Modeling](https://mlsanity.com/paper/2204.12681/) (2022)
- [Dynamically Retrieving Knowledge via Query Generation for Informative Dialogue Generation](https://mlsanity.com/paper/2208.00128/) (2022)
- [Contextual Knowledge Learning For Dialogue Generation](https://mlsanity.com/paper/2305.18200/) (2023)
- [Graph vs. Sequence: An Empirical Study on Knowledge Forms for Knowledge-Grounded Dialogue](https://mlsanity.com/paper/2312.07868/) (2023)
- [Blending Reward Functions via Few Expert Demonstrations for Faithful and Accurate Knowledge-Grounded Dialogue Generation](https://mlsanity.com/paper/2311.00953/) (2023)
- [UniRQR: A Unified Model for Retrieval Decision, Query, and Response Generation in Internet-Based Knowledge Dialogue Systems](https://mlsanity.com/paper/2401.06811/) (2024)
- [Policy-driven Knowledge Selection and Response Generation for Document-grounded Dialogue](https://mlsanity.com/paper/2410.15970/) (2024)

## Abstract (arXiv)

Document grounded generation is the task of using the information provided in a document to improve text generation. This work focuses on two different document grounded generation tasks: Wikipedia Update Generation task and Dialogue response generation. Our work introduces two novel adaptations of large scale pre-trained encoder-decoder models focusing on building context driven representation of the document and enabling specific attention to the information in the document. Additionally, we provide a stronger BART baseline for these tasks. Our proposed techniques outperform existing methods on both automated (at least 48% increase in BLEU-4 points) and human evaluation for closeness to reference and relevance to the document. Furthermore, we perform comprehensive manual inspection of the generated output and categorize errors to provide insights into future directions in modeling these tasks.