# Modeling Context in Answer Sentence Selection Systems on a Latency Budget

arXiv: https://arxiv.org/abs/2101.12093 · HTML: https://mlsanity.com/paper/2101.12093/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 16 references that are in the map.
- First posted on arXiv 28 Jan 2021.

## Builds on (most cited first, 11 of 11)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning](https://mlsanity.com/paper/1909.00277/) (2019)
- [TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection](https://mlsanity.com/paper/1911.04118/) (2019)
- [How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representations](https://mlsanity.com/paper/1909.04925/) (2019)
- [DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering](https://mlsanity.com/paper/2005.00697/) (2020)
- [Efficient and Robust Question Answering from Minimal Context over Documents](https://mlsanity.com/paper/1805.08092/) (2018)
- [The Cascade Transformer: an Application for Efficient Answer Sentence Selection](https://mlsanity.com/paper/2005.02534/) (2020)
- [Contextualized Representations Using Textual Encyclopedic Knowledge](https://mlsanity.com/paper/2004.12006/) (2020)

## Built on by (most cited first, 9 of 9)

- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Will this Question be Answered? Question Filtering via Answer Model Distillation for Efficient Question Answering](https://mlsanity.com/paper/2109.07009/) (2021)
- [Answer Generation for Retrieval-based Question Answering Systems](https://mlsanity.com/paper/2106.00955/) (2021)
- [Ensemble Transformer for Efficient and Accurate Ranking Tasks: an Application to Question Answering Systems](https://mlsanity.com/paper/2201.05767/) (2022)
- [Cross-Lingual Knowledge Distillation for Answer Sentence Selection in Low-Resource Languages](https://mlsanity.com/paper/2305.16302/) (2023)
- [Recording for Eyes, Not Echoing to Ears: Contextualized Spoken-to-Written Conversion of ASR Transcripts](https://mlsanity.com/paper/2408.09688/) (2024)
- [Context-Aware Transformer Pre-Training for Answer Sentence Selection](https://mlsanity.com/paper/2305.15358/) (2023)
- [Meeting Action Item Detection with Regularized Context Modeling](https://mlsanity.com/paper/2303.16763/) (2023)
- [Question-Context Alignment and Answer-Context Dependencies for Effective Answer Sentence Selection](https://mlsanity.com/paper/2306.02196/) (2023)

## Abstract (arXiv)

Answer Sentence Selection (AS2) is an efficient approach for the design of open-domain Question Answering (QA) systems. In order to achieve low latency, traditional AS2 models score question-answer pairs individually, ignoring any information from the document each potential answer was extracted from. In contrast, more computationally expensive models designed for machine reading comprehension tasks typically receive one or more passages as input, which often results in better accuracy. In this work, we present an approach to efficiently incorporate contextual information in AS2 models. For each answer candidate, we first use unsupervised similarity techniques to extract relevant sentences from its source document, which we then feed into an efficient transformer architecture fine-tuned for AS2. Our best approach, which leverages a multi-way attention architecture to efficiently encode context, improves 6% to 11% over noncontextual state of the art in AS2 with minimal impact on system latency. All experiments in this work were conducted in English.