# Contextualized Sparse Representations for Real-Time Open-Domain Question Answering

arXiv: https://arxiv.org/abs/1911.02896 · HTML: https://mlsanity.com/paper/1911.02896/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Nov 2019.

## Built on by (most cited first, 11 of 11)

- [Phrase Retrieval Learns Passage Retrieval, Too](https://mlsanity.com/paper/2109.08133/) (2021)
- [A Survey for Efficient Open Domain Question Answering](https://mlsanity.com/paper/2211.07886/) (2022)
- [Question Answering Infused Pre-training of General-Purpose Contextualized Representations](https://mlsanity.com/paper/2106.08190/) (2021)
- [UnifieR: A Unified Retriever for Large-Scale Retrieval](https://mlsanity.com/paper/2205.11194/) (2022)
- [Constructing Tree-based Index for Efficient and Effective Dense Retrieval](https://mlsanity.com/paper/2304.11943/) (2023)
- [Long-tailed Extreme Multi-label Text Classification with Generated Pseudo Label Descriptions](https://mlsanity.com/paper/2204.00958/) (2022)
- [Two-Step Question Retrieval for Open-Domain QA](https://mlsanity.com/paper/2205.09393/) (2022)
- [Neural Retriever and Go Beyond: A Thesis Proposal](https://mlsanity.com/paper/2205.16005/) (2022)
- [Bridging the Training-Inference Gap for Dense Phrase Retrieval](https://mlsanity.com/paper/2210.13678/) (2022)
- [Context Retrieval via Normalized Contextual Latent Interaction for Conversational Agent](https://mlsanity.com/paper/2312.00774/) (2023)
- [Towards Robust Evaluation: A Comprehensive Taxonomy of Datasets and Metrics for Open Domain Question Answering in the Era of Large Language Models](https://mlsanity.com/paper/2406.13232/) (2024)

## Abstract (arXiv)

Open-domain question answering can be formulated as a phrase retrieval problem, in which we can expect huge scalability and speed benefit but often suffer from low accuracy due to the limitation of existing phrase representation models. In this paper, we aim to improve the quality of each phrase embedding by augmenting it with a contextualized sparse representation (Sparc). Unlike previous sparse vectors that are term-frequency-based (e.g., tf-idf) or directly learned (only few thousand dimensions), we leverage rectified self-attention to indirectly learn sparse vectors in n-gram vocabulary space. By augmenting the previous phrase retrieval model (Seo et al., 2019) with Sparc, we show 4%+ improvement in CuratedTREC and SQuAD-Open. Our CuratedTREC score is even better than the best known retrieve & read model with at least 45x faster inference speed.