# EfficientQA : a RoBERTa Based Phrase-Indexed Question-Answering System

arXiv: https://arxiv.org/abs/2101.02157 · HTML: https://mlsanity.com/paper/2101.02157/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 24 references that are in the map.
- First posted on arXiv 6 Jan 2021.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [Bidirectional Attention Flow for Machine Comprehension](https://mlsanity.com/paper/1611.01603/) (2016)
- [End-to-End Open-Domain Question Answering with BERTserini](https://mlsanity.com/paper/1902.01718/) (2019)
- [Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Index](https://mlsanity.com/paper/1906.05807/) (2019)
- [FQuAD: French Question Answering Dataset](https://mlsanity.com/paper/2002.06071/) (2020)
- [Efficient and Robust Question Answering from Minimal Context over Documents](https://mlsanity.com/paper/1805.08092/) (2018)
- [Ranking Paragraphs for Improving Answer Recall in Open-Domain Question Answering](https://mlsanity.com/paper/1810.00494/) (2018)
- [Phrase-Indexed Question Answering: A New Challenge for Scalable Document Comprehension](https://mlsanity.com/paper/1804.07726/) (2018)
- [Accelerating Real-Time Question Answering via Question Generation](https://mlsanity.com/paper/2009.05167/) (2020)
- [R3: Reinforced Reader-Ranker for Open-Domain Question Answering](https://mlsanity.com/paper/1709.00023/) (2017)
- [Weaver: Deep Co-Encoding of Questions and Documents for Machine Reading](https://mlsanity.com/paper/1804.10490/) (2018)

## Built on by (most cited first, 2 of 2)

- [Sentiment analysis in tweets: an assessment study from classical to modern text representation models](https://mlsanity.com/paper/2105.14373/) (2021)
- [Question-Answering Model for Schizophrenia Symptoms and Their Impact on Daily Life using Mental Health Forums Data](https://mlsanity.com/paper/2310.00448/) (2023)

## Abstract (arXiv)

State-of-the-art extractive question-answering models achieve superhuman performances on the SQuAD benchmark. Yet, they are unreasonably heavy and need expensive GPU computing to answer questions in a reasonable time. Thus, they cannot be used in the open-domain question-answering paradigm for real-world queries on hundreds of thousands of documents. In this paper, we explore the possibility of transferring the natural language understanding of language models into dense vectors representing questions and answer candidates to make question-answering compatible with a simple nearest neighbor search task. This new model, which we call EfficientQA, takes advantage of the pair of sequences kind of input of BERT-based models to build meaningful, dense representations of candidate answers. These latter are extracted from the context in a question-agnostic fashion. Our model achieves state-of-the-art results in Phrase-Indexed Question Answering (PIQA), beating the previous state-of-art by 1.3 points in exact-match and 1.4 points in f1-score. These results show that dense vectors can embed rich semantic representations of sequences, although these were built from language models not originally trained for the use case. Thus, to build more resource-efficient NLP systems in the future, training language models better adapted to build dense representations of phrases is one of the possibilities.