Context-Aware Transformer Pre-Training for Answer Sentence Selection
Not on the map. Why?
Abstract (arXiv)
Answer Sentence Selection (AS2) is a core component for building an accurate Question Answering pipeline. AS2 models rank a set of candidate sentences based on how likely they answer a given question. The state of the art in AS2 exploits pre-trained transformers by transferring them on large annotated datasets, while using local contextual information around the candidate sentence. In this paper, we propose three pre-training objectives designed to mimic the downstream fine-tuning task of contextual AS2. This allows for specializing LMs when fine-tuning for contextual AS2. Our experiments on three public and two large-scale industrial datasets show that our pre-training approaches (applied to RoBERTa and ELECTRA) can improve baseline contextual AS2 accuracy by up to 8% on some datasets.
Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Builds on 21 of its 38 references that are in the map. First posted on arXiv 24 May 2023.
Builds on
21 of 38 references.
References · 21
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,779
- 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,042
- 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 693
- 2021Datasets: A Community Library for Natural Language Processingcited by 529
- 2016NewsQA: A Machine Comprehension Datasetcited by 330
- 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
- 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
- 2021End-to-End Training of Neural Retrievers for Open-Domain Question Answeringcited by 75
- 2019TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selectioncited by 74
- 2021Few-Shot Question Answering by Pretraining Span Selectioncited by 62
9 more
- 2021Should We Be Pre-training? An Argument for End-task Aware Training as an Alternativecited by 26
- 2019A Compare-Aggregate Model with Latent Clustering for Answer Selectioncited by 19
- 2021Will this Question be Answered? Question Filtering via Answer Model Distillation for Efficient Question Answeringcited by 19
- 2016Contextual LSTM (CLSTM) models for Large scale NLP taskscited by 15
- 2021Modeling Context in Answer Sentence Selection Systems on a Latency Budgetcited by 9
- 2022Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selectioncited by 9
- 2022Paragraph-based Transformer Pre-training for Multi-Sentence Inferencecited by 7
- 2022Knowledge Transfer from Answer Ranking to Answer Generationcited by 6
- 2020A Study on Efficiency, Accuracy and Document Structure for Answer Sentence Selectioncited by 5
Built on by
Citing papers per year: 2023: 1.
1 paper.