# Neural Retrieval for Question Answering with Cross-Attention Supervised Data Augmentation

arXiv: https://arxiv.org/abs/2009.13815 · HTML: https://mlsanity.com/paper/2009.13815/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Sep 2020.

## Built on by (most cited first, 18 of 18)

- [RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking](https://mlsanity.com/paper/2110.07367/) (2021)
- [Data Augmentation Approaches in Natural Language Processing: A Survey](https://mlsanity.com/paper/2110.01852/) (2021)
- [PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrieval](https://mlsanity.com/paper/2108.06027/) (2021)
- [ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrieval](https://mlsanity.com/paper/2205.09153/) (2022)
- [A Sketch Is Worth a Thousand Words: Image Retrieval with Text and Sketch](https://mlsanity.com/paper/2208.03354/) (2022)
- [REBAR: Retrieval-Based Reconstruction for Time-series Contrastive Learning](https://mlsanity.com/paper/2311.00519/) (2023)
- [Sentence-aware Contrastive Learning for Open-Domain Passage Retrieval](https://mlsanity.com/paper/2110.07524/) (2021)
- [Retrieval Enhanced Data Augmentation for Question Answering on Privacy Policies](https://mlsanity.com/paper/2204.08952/) (2022)
- [ConTEXTual Net: A Multimodal Vision-Language Model for Segmentation of Pneumothorax](https://mlsanity.com/paper/2303.01615/) (2023)
- [Measuring and Addressing Indexical Bias in Information Retrieval](https://mlsanity.com/paper/2406.04298/) (2024)
- [Policy-Gradient Training of Language Models for Ranking](https://mlsanity.com/paper/2310.04407/) (2023)
- [NAPG: Non-Autoregressive Program Generation for Hybrid Tabular-Textual Question Answering](https://mlsanity.com/paper/2211.03462/) (2022)
- [Adam: Dense Retrieval Distillation with Adaptive Dark Examples](https://mlsanity.com/paper/2212.10192/) (2022)
- [Recent Advances in Automated Question Answering In Biomedical Domain](https://mlsanity.com/paper/2111.05937/) (2021)
- [Retrieval-Augmented Classification with Decoupled Representation](https://mlsanity.com/paper/2303.13065/) (2023)
- [Hybrid and Collaborative Passage Reranking](https://mlsanity.com/paper/2305.09313/) (2023)
- [Investigating Masking-based Data Generation in Language Models](https://mlsanity.com/paper/2307.00008/) (2023)
- [Query-oriented Data Augmentation for Session Search](https://mlsanity.com/paper/2407.03720/) (2024)

## Abstract (arXiv)

Neural models that independently project questions and answers into a shared embedding space allow for efficient continuous space retrieval from large corpora. Independently computing embeddings for questions and answers results in late fusion of information related to matching questions to their answers. While critical for efficient retrieval, late fusion underperforms models that make use of early fusion (e.g., a BERT based classifier with cross-attention between question-answer pairs). We present a supervised data mining method using an accurate early fusion model to improve the training of an efficient late fusion retrieval model. We first train an accurate classification model with cross-attention between questions and answers. The accurate cross-attention model is then used to annotate additional passages in order to generate weighted training examples for a neural retrieval model. The resulting retrieval model with additional data significantly outperforms retrieval models directly trained with gold annotations on Precision at N (P@N) and Mean Reciprocal Rank (MRR).