# AttenWalker: Unsupervised Long-Document Question Answering via Attention-based Graph Walking

arXiv: https://arxiv.org/abs/2305.02235 · HTML: https://mlsanity.com/paper/2305.02235/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 3 May 2023.

## Builds on (most cited first, 15 of 15)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [Bidirectional Attention Flow for Machine Comprehension](https://mlsanity.com/paper/1611.01603/) (2016)
- [A guide to convolution arithmetic for deep learning](https://mlsanity.com/paper/1603.07285/) (2016)
- [Multilingual Constituency Parsing with Self-Attention and Pre-Training](https://mlsanity.com/paper/1812.11760/) (2018)
- [Unsupervised Question Answering by Cloze Translation](https://mlsanity.com/paper/1906.04980/) (2019)
- [Unsupervised Multi-hop Question Answering by Question Generation](https://mlsanity.com/paper/2010.12623/) (2020)
- [Harvesting and Refining Question-Answer Pairs for Unsupervised QA](https://mlsanity.com/paper/2005.02925/) (2020)
- [Long Context Question Answering via Supervised Contrastive Learning](https://mlsanity.com/paper/2112.08777/) (2021)
- [Capturing Global Structural Information in Long Document Question Answering with Compressive Graph Selector Network](https://mlsanity.com/paper/2210.05499/) (2022)
- [Unsupervised Question Answering via Answer Diversifying](https://mlsanity.com/paper/2208.10813/) (2022)

## Built on by (most cited first, 3 of 3)

- [Multi-view Content-aware Indexing for Long Document Retrieval](https://mlsanity.com/paper/2404.15103/) (2024)
- [MemSum-DQA: Adapting An Efficient Long Document Extractive Summarizer for Document Question Answering](https://mlsanity.com/paper/2310.06436/) (2023)
- [Evidence-Enhanced Triplet Generation Framework for Hallucination Alleviation in Generative Question Answering](https://mlsanity.com/paper/2408.15037/) (2024)

## Abstract (arXiv)

Annotating long-document question answering (long-document QA) pairs is time-consuming and expensive. To alleviate the problem, it might be possible to generate long-document QA pairs via unsupervised question answering (UQA) methods. However, existing UQA tasks are based on short documents, and can hardly incorporate long-range information. To tackle the problem, we propose a new task, named unsupervised long-document question answering (ULQA), aiming to generate high-quality long-document QA instances in an unsupervised manner. Besides, we propose AttenWalker, a novel unsupervised method to aggregate and generate answers with long-range dependency so as to construct long-document QA pairs. Specifically, AttenWalker is composed of three modules, i.e., span collector, span linker and answer aggregator. Firstly, the span collector takes advantage of constituent parsing and reconstruction loss to select informative candidate spans for constructing answers. Secondly, by going through the attention graph of a pre-trained long-document model, potentially interrelated text spans (that might be far apart) could be linked together via an attention-walking algorithm. Thirdly, in the answer aggregator, linked spans are aggregated into the final answer via the mask-filling ability of a pre-trained model. Extensive experiments show that AttenWalker outperforms previous methods on Qasper and NarrativeQA. In addition, AttenWalker also shows strong performance in the few-shot learning setting.